Experiments Outline

This note records the evolving structure for path:paper/conference-paper/sections/04-experiments.tex. The experiments section should establish the controlled evaluation setting, then present the empirical findings that support the paper's central claim: base-directed reconstruction can learn strategy-aligned router-generator factorizations of a base-model response distribution while standard conditional ELBO training can preserve observable behavior without recovering a useful latent variable.

The first settled subsection outline is Benchmark and Evaluation Protocol. This subsection should be compact in the NeurIPS main text, likely around one third to one half page, with data-generation, validator, split, and implementation details deferred to appendix.

Related concept notes:

• Findings
• Task data and evaluation setup
• Metrics and frontier language
• Cross-input semantic consistency
• Strategy alignment and recovery
• Latent strategy variable

Related implementation references:

• path:experiments/synthetic_sequences/tasks/README.md
• path:experiments/synthetic_sequences/tasks/multi_task.py
• path:experiments/synthetic_sequences/disentangled/docs/supervision_and_eval.md
• path:experiments/synthetic_sequences/disentangled/docs/metrics.md

Resolved Setup Decisions

• Task-family presentation: Default to an inline list of the six task families in main text, with the full task/strategy table in appendix. Keep a compact main-text table draft available for a less page-constrained arXiv version or if the 9-page version has room.
• Primary factorization training source: Foreground model-generated samples from the frozen base model: $x\sim\mathcal D_X$ and $y\sim p_\theta(\cdot|x)$. This best matches the paper's framing as factorizing a base-model response distribution rather than training directly against benchmark labels.
• Model regimes: Describe both pretrained and random-initialized regimes without over-demarcating their roles in the setup prose. In both cases, the model is first fit or adapted to the benchmark distribution so that it captures a multi-strategy problem-solving response distribution with implicit strategy information and no explicit strategy variable. The router-generator methodology is then applied to factorize that response distribution. Figure captions can specify whether each result uses pretrained or random-initialized runs. The current narrative plan is for headline success figures, such as method metric bars, to use the larger-scale pretrained regime, while denser diagnostic analyses can use larger random-initialized sweeps.
• Distributional Fidelity: In the main experiments, Distributional Fidelity should denote the probability that a generated response from the factorized model is a valid solution mapping to one of the reference solution strategies: $x\sim\mathcal D_X$, $y\sim p_\phi(\cdot|x)$, and $y$ is valid and strategy-attributable. The exact estimator and treatment of parse/validity categories can go in appendix.
• Analogical Consistency: Invalid traces count as failures. The main metric is $\Pr[\operatorname{strat}(Y)=\operatorname{strat}(Y')]$ under the analogical transfer protocol, with missing or invalid strategy attributions treated as non-agreement.
• Router Strategy Decodability: Mention Router Strategy Decodability as a supporting diagnostic when describing figures that use it, but make Analogical Consistency the main metric behind the compact Strategy Alignment label.

Benchmark And Evaluation Protocol

Purpose And Scope

This subsection should give the reader enough experimental context to understand F1–F6 without turning into a data-generation section. It should define the controlled benchmark, distinguish base-model fitting from factorization training, state that strategy annotations are withheld from factorization training, and preview the metrics used by the result figures.

The subsection should answer four reader questions:

1. What is the controlled setting in which strategy alignment can be measured?
2. What distribution is used to fit the base model, and what distribution is used to train the factorization?
3. What information is available only for evaluation?
4. What empirical properties do the main metrics estimate?

Main-Text Paragraph Skeleton

Paragraph 1: Why the benchmark exists. Introduce the controlled multi-strategy algorithmic benchmark at first mention, then use algorithmic strategy benchmark as the short form. Explain that open-ended language modeling does not provide direct ground truth for strategy alignment, so the controlled benchmark supplies problem instances, solution traces, and reference strategy annotations. State that these annotations make it possible to measure strategy alignment and cross-input semantic consistency.

Reference content:

We evaluate on a controlled multi-strategy algorithmic benchmark: a suite of task families in which each problem instance admits multiple valid solution strategies with reference strategy annotations. This setting lets us evaluate whether a learned latent variable aligns with strategy-level variation in solution traces and whether that alignment is stable across related inputs.

Paragraph 2: What the benchmark contains. Define the core objects: a problem instance $x$, a solution trace $y$, and a solution strategy. Mention the six production task families in one compact sentence unless a main-text table is later approved: list summation, sorting algorithms, grid pathfinding, linear equation solving, base conversion, and multidigit addition. State that generated traces can be parsed, validated, and assigned a strategy attribution when they match a benchmark validator.

Reference content:

Each example consists of a problem instance $x$ and a solution trace $y$. In the controlled benchmark, solution traces instantiate one of several task-specific solution strategies; validators parse generated traces and infer a strategy attribution when a trace is valid and uniquely matched.

Paragraph 3: Two-stage experimental protocol. Distinguish the task/data distribution $\mathcal D$ from the frozen base-model response distribution $p_\theta(y|x)$. First, a language model is fit or adapted to the benchmark task/data distribution, producing a strategy-entangled base model. Then $p_\theta$ is frozen and the router-generator factorization is trained against the base-model response distribution, with $x\sim\mathcal D_X$ and $y\sim p_\theta(\cdot|x)$. State explicitly that strategy annotations are withheld from factorization training and used only for evaluation.

Candidate display:

$$(x,y)\sim\mathcal D \quad\text{for base-model fitting},\qquad x\sim\mathcal D_X,\; y\sim p_\theta(\cdot|x) \quad\text{for factorization training}.$$

Paragraph 4: Metrics preview. Define only the metrics needed to read the result figures. Distributional Fidelity measures preservation of observable behavior. Strategy Alignment is the compact main-figure label for cross-input latent strategy alignment when measured by Analogical Consistency. Router Strategy Decodability is a weaker probe diagnostic for strategy information in router-sampled latents, if it appears in main text or appendix. Base-relative reconstruction $\mathcal R_\theta$ measures the fraction of z-free base-model surprisal remaining after conditioning on $z$, and $1-\mathcal R_\theta$ is residual surprise explained by $z$.

Reference content:

We report both behavior-preservation and latent-alignment metrics. Distributional Fidelity is the probability that a response generated from the factorized model is a valid, strategy-attributable solution. Strategy Alignment denotes cross-input latent strategy alignment, measured by Analogical Consistency under an analogical transfer intervention. We also use base-relative reconstruction $\mathcal R_\theta$ and its complement $1-\mathcal R_\theta$ to analyze how much residual base-model surprise is explained by $z$.

Main-Text Space Budget

The main-text subsection should probably be four compact paragraphs. A task-family table is useful but may be too expensive under the 9-page limit. If included, it should be a very small table with columns for task family, example strategy names, and evaluation role; otherwise, the full table should move to appendix.

Optional Main-Text Task Table Draft

This table is not required for the 9-page submission. Keep it as an optional arXiv/full-version artifact or a fallback if the main text has room.

Appendix Detail Targets

The appendix should carry details that are necessary for reproducibility but not for the main narrative:

• full task-family descriptions;
• strategy label lists for each task family;
• data-generation parameters and ranges;
• train/validation/test split sizes;
• sequence formatting and tokenization;
• validators and strategy-attribution rules;
• base-model fitting details;
• factorization supervision source details, including ground-truth versus model-generated supervision if both are used;
• exact metric estimators;
• model sizes, adaptation modes, seeds, and hyperparameter grids.

Missing Information To Resolve Before Final Prose

1. Exact main-experiment task settings: parameter ranges, sequence lengths, and whether all six task families appear in every main figure.
2. Dataset sizes and split sizes.
3. Which figures use random-initialized models versus pretrained Qwen2.5 runs.
4. Confirm from figure artifacts which main results train factorization on model-generated samples from $p_\theta$ versus any ground-truth-trace variants.
5. Exact implementation estimator for Distributional Fidelity, including parse/validity/strategy-attribution category handling.
6. Exact Analogical Consistency estimator: source/target sampling and same-task-family restriction. Invalid traces count as failures by decision.
7. Which figures expose Router Strategy Decodability in main text, versus appendix-only diagnostics.

Current Experiments Narrative Direction

The intended finding order is: first establish F1 — Implicit Strategy Structure In The Base Model, then present F2 — ELBO Under-Specification And Collapse, then show F3 — Strategy-Aligned Router-Generator Factorization, followed by F6 — Cross-Input Semantic Consistency and F5 — Alignment-Reconstruction Frontier or mechanism/frontier analysis. This ordering keeps the benchmark and evaluation setup before the empirical findings that depend on reference strategy annotations.

Approved Visible Subsection Structure

Use three visible subsections in the experiments section:

1. Benchmark and evaluation protocol
2. Recovering Strategy-Aligned Latents
3. How Model-Directed Reconstruction Pressure Drives Strategy Alignment

The first subsection introduces the benchmark, two-stage protocol, and compact metric definitions. The second subsection is the main empirical results subsection: it should fold F1 into an opening paragraph, fold F2 into the baseline contrast, present F3 as the main success result, and include F6 as a short \paragraph{Cross-input semantic consistency.}. The third subsection carries the mechanism/frontier analysis and F5.

Within Recovering Strategy-Aligned Latents, use paragraph labels only where they improve scanability, for example:

• \paragraph{Implicit strategy information.} for F1 and fig:entanglement;
• \paragraph{ELBO failure.} for F2 and the reference back to fig:elbo-fails;
• \paragraph{Cross-input semantic consistency.} for F6 and fig:analogical-consistency.

Section-Level Prose Outline

This outline records paragraph-level logic, not exact prose or fixed figure placement. Figure placement can be adjusted during LaTeX drafting.

Benchmark and evaluation protocol

Purpose: Establish the controlled setting and metric vocabulary needed to interpret all experiment findings.

Reader question: How do we measure whether a latent-variable factorization is strategy-aligned rather than merely predictive?

Paragraph flow:

1. Introduce the controlled multi-strategy algorithmic benchmark and state why it is needed: open-ended language modeling lacks observable strategy labels, while the controlled benchmark provides problem instances, solution traces, and reference strategy annotations.
2. Define the benchmark objects: problem instance $x$, solution trace $y$, solution strategy, and reference strategy annotation. Mention the six task families inline, with full task/strategy details deferred to appendix.
3. Explain the two-stage protocol: first fit or adapt a base model to the benchmark distribution so it represents a multi-strategy response distribution; then freeze $p_\theta$ and train the router-generator factorization on $x\sim\mathcal D_X$, $y\sim p_\theta(\cdot|x)$. State explicitly that strategy annotations are withheld from factorization training and used only for evaluation.
4. Preview metrics: Distributional Fidelity, Strategy Alignment as Analogical Consistency, Router Strategy Decodability as supporting diagnostic, base-relative reconstruction $\mathcal R_\theta$, and residual surprise explained by $z$ as $1-\mathcal R_\theta$.

Claims to avoid: Do not imply the benchmark uniquely captures all possible reasoning strategies. Do not imply factorization training uses strategy labels. Do not imply Distributional Fidelity is full distribution matching unless the estimator supports that.

Bridge to next subsection: The setup makes it possible to ask whether the base model already contains strategy information and whether a factorization method can expose it as a usable latent variable.

Recovering Strategy-Aligned Latents

Purpose: Present the main empirical results: strategy information is present in the base model; standard variational training does not expose it; model-directed reconstruction recovers strategy-aligned latents while maintaining Distributional Fidelity.

Reader question: Does the proposed method actually recover useful strategy-aligned latents?

Paragraph flow:

1. Open with the premise result: before factorization, strategy information is decodable from base-model hidden states. This supports find:implicit-strategy-structure and fig:entanglement. Emphasize that this shows implicit strategy information, not a usable latent factorization.
2. Contrast with standard variational training: even though the strategy information is present, standard ELBO-style training can maintain Distributional Fidelity while leaving $z$ inert or unaligned. This supports find:elbo-under-specified, references the methodology ELBO argument and fig:elbo-fails, and connects to the method comparison bars.
3. State the main success result: model-directed reconstruction pressure, through the formal $p_\theta$-directed reconstruction objective, recovers strategy-aligned latents while preserving valid task behavior. This supports find:strategy-aligned-factorization, fig:disentanglement, and fig:reconstruction-decodability.
4. Interpret the latent-geometry evidence: the learned latent structure separates reference strategies in a way consistent with the intended role of a latent strategy variable. Keep this tied to the controlled benchmark and reference annotations.
5. Interpret the method comparison: proposed variants maintain Distributional Fidelity while achieving substantially stronger Strategy Alignment than standard variational objective baselines.
6. \paragraph{Cross-input semantic consistency.} Explain that local strategy separation is not enough; the same latent should carry stable strategy meaning across related inputs. Define Analogical Consistency as the evaluation of this property and use find:cross-input-semantic-consistency. Note that fig:analogical-consistency is a definition/evaluation schematic, while fig:reconstruction-decodability supplies the quantitative evidence.
7. State the significance of cross-input consistency in surrounding prose: it is the property that makes latent-space strategy steering plausible and suggests future uses in directed exploration, recombination of model behaviors, and interpretability. Keep this framed as implication, not evaluated downstream application.

Claims to avoid: Do not claim the latent recovers a unique true strategy ontology beyond the controlled benchmark. Do not claim downstream steering, RL exploration, or ability recombination is evaluated. Do not claim ELBO fails to fit observable behavior.

Bridge to next subsection: Having shown that the method succeeds, the next question is what aspect of the objective explains the alignment gains across methods and baselines.

How Model-Directed Reconstruction Pressure Drives Strategy Alignment

Purpose: Explain the mechanism evidence: strategy alignment tracks token-weighted, model-directed reconstruction pressure across methods and baselines.

Reader question: Why does the proposed objective recover strategy-aligned latents when standard variational objectives fail?

Paragraph flow:

1. Open by shifting from outcome to mechanism: the main results show that model-directed reconstruction works, but the diagnostic question is whether alignment improves when $z$ is pressured to explain the $p_\theta$-directed, strategy-relevant parts of the response.
2. Define the diagnostic axis in prose: token-weighted reconstruction loss reflects whether the model is explaining high base-surprisal response regions emphasized by the $p_\theta$-directed objective. Connect this to the methodology section's token-level $p_\theta$ surprisal weighting.
3. Present the main mechanism claim: token-weighted reconstruction loss tracks strategy alignment across both proposed methods and variational baselines. This supports find:alignment-reconstruction-frontier.
4. Interpret baseline failure: ELBO-style objectives can fit observable behavior without moving reconstruction pressure into the strategy-relevant residual regions where $z$ becomes useful.
5. Interpret proposed-method success: model-directed reconstruction pressure improves alignment because it makes the latent variable explain response variation that is difficult for the z-free base model and relevant to strategy choices.
6. End with a bridge to theory: the empirical mechanism suggests that success depends not only on latent-variable expressivity, but on objective terms that select among marginal-preserving factorizations; the following theory section studies conditions under which objective selection recovers strategy structure.

Claims to avoid: Do not overstate the diagnostic as a causal proof. Say "tracks," "supports the mechanism," or "supports the view that," rather than claiming the scatter alone proves causality. Do not imply token-weighted reconstruction is the target metric; it is a mechanistic diagnostic connected to the objective.

Approved Finding Box Set

Keep five boxed findings in path:paper/conference-paper/sections/04-experiments.tex, using the current LaTeX labels as the source of truth rather than the provisional numbering in Findings.

1. find:implicit-strategy-structure: keep as the premise box. It should state that strategy information is decodable from base-model hidden states before factorization.
2. find:elbo-under-specified: keep boxed. It is a core contrast finding, supported both by fig:elbo-fails and by the method comparison bars: standard variational training can preserve observable behavior while failing to recover a strategy-aligned latent.
3. find:strategy-aligned-factorization: keep and prioritize as the central success box. It should focus on latent geometry and method comparison figures, stating that model-directed reconstruction pressure maintains Distributional Fidelity while achieving strategy alignment and strategy separation.
4. find:cross-input-semantic-consistency: keep as a finer interpretability/control finding. The quantitative evidence is Analogical Consistency in fig:reconstruction-decodability; fig:analogical-consistency explains the evaluation intervention. Phrase conservatively in terms of cross-input semantic consistency and analogical strategy transfer; use steerability/control language only when tied directly to the shared-$z$ intervention.
5. find:alignment-reconstruction-frontier: keep, but revise toward the sharper mechanism claim: token-weighted, model-directed reconstruction pressure closely tracks strategy alignment across proposed and baseline methods, explaining ELBO/baseline failure as a failure to drive reconstruction pressure toward $p_\theta$-directed, strategy-relevant response regions.

Each box should be compact, ideally one tight paragraph of no more than two sentences, and should be attached to nearby figure evidence rather than repeating surrounding prose.

Finding Box Wording Policy

• Use model-directed reconstruction pressure as the main phrase in empirical finding prose when emphasizing the mechanism and training pressure.
• Use $p_\theta$-directed reconstruction objective when naming the formal objective. Use base-directed reconstruction only where the surrounding methodology terminology needs continuity.
• For the central success title, use strategy-aligned factorizations or strategy-aligned latents. Use recovers strategy structure only in the body when tied to reference strategy annotations and figures.
• Use maintaining Distributional Fidelity in finding boxes, with the setup subsection defining Distributional Fidelity as valid, strategy-attributable generated solutions.
• For cross-input claims, title with cross-input semantic consistency and mention analogical strategy transfer in the body. Use steerability/control language only when directly tied to the shared-$z$ intervention.
• For the mechanism box, title with token-weighted reconstruction tracks strategy alignment. In the body, use causal language carefully, for example: "This supports the view that model-directed reconstruction pressure drives strategy alignment by…"

Approved Implicit Strategy Box Draft

For find:implicit-strategy-structure, use this wording as the working draft:

The base model represents implicit strategy information. Strategy information is decodable from the base model's hidden states before factorization. This suggests that the target structure is present in the model but not exposed as a usable latent variable.

Approved Central Success Box Draft

For find:strategy-aligned-factorization, use this wording as the working draft:

Model-directed reconstruction recovers strategy-aligned latents. The proposed $p_\theta$-directed reconstruction objective preserves valid task behavior while recovering latent structure aligned with reference solution strategies. This alignment is not achieved by standard variational objective baselines.

Approved ELBO Failure Box Draft

For find:elbo-under-specified, use this wording as the working draft:

ELBO training preserves behavior without identifying strategy. Standard variational objectives can maintain Distributional Fidelity while leaving $z$ inert or unaligned with reference solution strategies. This shows that fitting the observable response distribution is not sufficient to select a strategy-aligned factorization.

Approved Cross-Input Semantic Consistency Box Draft

For find:cross-input-semantic-consistency, use this wording as the working draft:

Recovered latents are semantically consistent across inputs. Our method recovers latents with cross-input semantic consistency, not merely within-input strategy separation. Analogical Consistency measures this property by testing whether reusing the same $z$ transfers the generated solution strategy from $X$ to $X'$.

The finding box should stay evidence-grounded. The broader significance should appear in surrounding prose rather than inside the box: cross-input consistency matters because a strategy-aligned latent is useful only if its meaning can be reused beyond the input on which it was inferred. It is the property that makes latent-space strategy steering plausible: a latent direction or region can be selected on one problem and applied analogically to another. Downstream applications such as directed exploration, controlled recombination of model behaviors, and interpretable latent descriptions of response variation should be framed as implications or future uses, not evaluated claims.

Approved Mechanism Box Draft

For find:alignment-reconstruction-frontier, use this wording as the working draft:

Model-directed reconstruction pressure explains alignment gains. Token-weighted reconstruction loss tracks strategy alignment across both proposed methods and variational baselines. This supports the mechanism that alignment improves when $z$ is pressured to explain response regions where the base model has strategy-relevant residual surprisal.

Findings And Figure Narrative

F1: Implicit Strategy Information In The Base Model

Role: Establish that the target structure is present before factorization.

Primary figure: fig:entanglement.

Narrative point: Strategy annotations are decodable from base-model hidden states, so the benchmark target is not absent from the model. This supports the premise that a factorization might expose strategy structure already implicit in the base-model response distribution.

F2: Standard ELBO Does Not Recover A Strategy-Aligned Latent

Role: Empirically connect the methodology section's ELBO non-identification argument to the benchmark.

Primary figure: fig:elbo-fails, placed in methodology as a diagnostic preview and referred back to after the benchmark/metric setup is defined.

Narrative point: Standard conditional ELBO can preserve observable behavior while leaving $z$ inert or unaligned. Experiments should avoid implying that ELBO fails to model $p_\theta(y|x)$; the failure is that marginal preservation does not select the intended latent factorization.

F3: Base-Directed Reconstruction Learns Strategy-Aligned Factorizations

Role: Main success result.

Primary figures: fig:disentanglement for latent-geometry/dynamics evidence and fig:reconstruction-decodability for quantitative method comparison.

Narrative point: Base-directed reconstruction learns router-generator factorizations whose latents are strategy-aligned while preserving valid task behavior. The headline success figure can use the larger-scale pretrained regime, while denser dynamics and diagnostics may use larger random-initialized sweeps.

Cross-Input Semantic Consistency Paragraph

Placement decision: Keep this as a short \paragraph{Cross-input semantic consistency.} inside the main success subsection rather than as a separate visible subsection.

Primary quantitative evidence: The Strategy Alignment metric in fig:reconstruction-decodability, defined as cross-input latent strategy alignment measured by Analogical Consistency.

Definition figure: fig:analogical-consistency is not an experimental-results figure. It depicts the analogical transfer evaluation used to define Analogical Consistency: a latent selected or inferred on a source input is reused on a related target input, and the generated strategies are compared. Use it to explain the metric, while the bar plot demonstrates empirical success and baseline failure on this metric.

Narrative point: The method should not merely separate strategies within a single input; it should recover latents whose strategy meaning is stable across related inputs. Analogical Consistency evaluates this stronger cross-input semantic consistency, and the main metric bar plot shows that the proposed method succeeds where baselines fail.

F5: Token-Weighted Reconstruction Tracks Strategy Alignment

Role: Mechanism/frontier analysis explaining why methods succeed or fail.

Primary figures: fig:fidelity-alignment-phase-space and fig:token-reconstruction-strategy-alignment-scatter, with possible support from fig:reconstruction-decodability.

Narrative point: The key claim is not merely a generic alignment-reconstruction frontier. The stronger mechanism claim is that the proposed objective's token-weighted reconstruction loss closely tracks strategy alignment across both proposed methods and baseline methods. ELBO baseline failure is explained by the fact that standard ELBO does not drive reconstruction pressure toward the $p_\theta$-directed regions where $z$ can explain residual strategy-relevant variation. Use token-level $p_\theta$ surprisal weighting and base-directed reconstruction pressure language from the methodology section.

Wording caution: Avoid suggesting that token-weighted reconstruction is itself the target metric. It is a mechanistic diagnostic: when training pressure makes $z$ explain the $p_\theta$-directed, strategy-relevant parts of the response, strategy alignment improves.