continuous_cvae-Apr24-reconstruction-stack_20260424 Analysis

Back to results

Current Reading

The Apr24 collection strengthens the evidence that reconstruction scaling is central to latent-strategy decodability in the continuous-CVAE synthetic setting. Across the completed 234-job grid, every normalized reconstruction variant has a higher mean matched strategy router score than every raw ELBO baseline. The raw baselines retain near-ceiling final answer accuracy, but their router probe and posterior probe metrics are much lower, so they do not provide the same evidence for recoverable strategy routing.

The strongest aggregate setting is dist_plus_tokcc_a0p5_b0p01_d50_lin50, with mean matched strategy router score 0.8532. The next settings are also normalized variants that combine distribution-level normalization with token surprisal-share weighting or the equal three-way stack. This suggests the Apr22 beta-schedule story is incomplete by itself: delayed beta helps, but the reconstruction-loss scale and weighting scheme appear to be a major part of the effect.

Token surprisal-share weighting has positive paired contrasts over distribution-only normalization in this sweep. The largest paired contrast is tok_a0p5_minus_dist_d50 at +0.0347 mean matched-score delta. The alpha=1.0 variant is also positive on mean, but the aggregate ranking favors alpha=0.5, especially under the delayed-50 schedule.

Example-level inverse-surprisal normalization is competitive, but its profile is less uniformly attractive than the top token-weighted variants. It reaches high matched score, especially under the delayed-50 schedule, but the same row has lower mean final answer accuracy than the top token-weighted settings.

Adaptation mode remains a real axis of variation. full and lora are close on paired matched score, with a small negative mean for lora - full but a small positive median. reinit is materially lower than both on matched score and final answer accuracy, while router probe alone does not show the same penalty. This points to the composite score being sensitive to answer/strategy-match behavior, not only latent probe decodability.

Claim Map

Strongly supported by this collection:

• Raw ELBO-style reconstruction with beta=1 solves the tasks but does not produce the same router-side latent-strategy decodability as normalized reconstruction variants.
• Distribution-level and example/token-normalized reconstruction losses move runs onto a lower reconstruction-ratio and higher matched-score frontier.
• Matched strategy router score is the right primary metric for this comparison, because raw baselines have high final answer accuracy and strategy match rate but low router probe.

Moderately supported:

• Token surprisal-share weighting improves over distribution-only normalization, with the clearest evidence for alpha=0.5 under delayed-50 beta.
• Delayed-50 linear beta schedules help several normalized variants, especially the equal three-way stack.
• Full and LoRA adaptation are broadly comparable for this collection, while reinitialization is worse on the composite metric.

Still underdetermined:

• Whether alpha=0.5 is genuinely better than alpha=1.0 across families, or whether the aggregate ranking is driven by a subset of cells.
• Whether example-level inverse-surprisal normalization is a generally useful independent ingredient or mainly a useful member of the combined stack.
• Whether the raw baseline failure is caused by raw reconstruction scale, beta=1, KL dynamics, posterior collapse, or interaction between those factors.

Caveats

The Apr24 grid is not a full factorial isolation of every loss ingredient, beta value, and schedule. The raw baselines use beta=1, while the normalized variants use beta=0.01, so the raw-vs-normalized contrast should not be read as a pure reconstruction-normalization causal estimate without follow-up controls.

The sweep uses one seed per cell. Paired contrasts are informative because they match family and adaptation cells, but they are not seed-robust uncertainty estimates.

The composite metric intentionally mixes router probe with strategy match rate. That is useful for the Apr24 question, but follow-up diagnostics should continue reporting router probe, strategy match rate, final answer accuracy, posterior probe, reconstruction ratio, and KL separately.

Recommended Next Analyses

1. Split the paired contrast ladder by family and adaptation mode.
2. Add bootstrap or paired sign summaries for the key contrasts, especially token weighting and delayed schedules.
3. Inspect top/bottom latent snapshots family by family, focusing on raw baselines and the best normalized settings.
4. Run a raw low-beta control and a normalized higher-beta control to separate reconstruction scaling from beta magnitude.
5. Convert the strongest supported claims into writing-facing figure captions for the reconstruction-decoding frontier and initialization/adaptation figures.