Paper-Selected Reconstruction-Stack Losses: Results

Back to hub

This results note records the Apr27 paper basic-grid snapshot for exp2_continuous_cvae-Apr27-paper-basic-grid, extended with the Apr30 tokinv_a0p25 / dist0p05_tokcc0p95_a0p25 runs, the May01 exp2_continuous_cvae-May01-tokeninv-stress-dist-tokcc-followup collection, and the May02 exp2_continuous_cvae-May02-dist0p75-tokcc0p25-followup collection. It is observational only. Interpretive claims and paper-positioning recommendations are in analysis.

Collection State

The expected grid now has 504 cells: the 294-cell Apr27+Apr30 grid, 168 May01 follow-up cells, and 42 May02 dist0p75_tokcc0p25_a0p25 follow-up cells. The family-level and paper-facing summaries below use all 432 complete single-task runs. The artifact inventory also contains 72 complete multi_task cells; those are analyzed in the refreshed multitask report for the same Apr27+Apr30+May01+May02 collection coverage. Analogical summaries are complete for every complete artifact in the expected grid.

Status counts over the full expected grid:

Incomplete cells by task: none.

Loss-Setting Summary

Rows are means over complete single-task datasets. The primary metric is analysis/router_probe_times_strategy_match_rate; the analogical metric is analogical_steering/pair_set_overlap_rate.

The lowest-ranked rows by the same metric are:

May01 And May02 Follow-Up Checks

The May01 follow-up adds tokinv_a1_eps1em8, tokinv_a1_eps1em4, dist0p10_tokcc0p90_a0p25, and dist0p25_tokcc0p75_a0p25. The May02 follow-up adds dist0p75_tokcc0p25_a0p25. Both follow-up slices are complete, including analogical summaries. In the fixed b0p01/lin100 single-task comparison, the matched-strategy router scores are 0.8870 for the 0.25 distribution-normalized / 0.75 TOK-CC mixture, 0.8825 for the May02 0.75 / 0.25 mixture, and 0.8811 for pure TOK-CC; the corresponding analogical overlaps are 0.9295, 0.9316, and 0.9152.

Selected rows for the current paper-facing comparison:

Best fixed beta/schedule setting within each May01/May02 follow-up family:

Which Loss Family Is Best?

Each row below represents the best fixed beta/schedule setting within one reconstruction family, using the mean matched-strategy router score across the six complete single-task datasets. The standard deviation and worst-task score report across-task reliability.

Family, Beta, And Schedule Views

The family-by-beta/schedule heatmap shows the same single-task mean aggregation as the table above. The final-answer-accuracy heatmap is included as a guardrail view.

Beta Robustness

The beta-robustness panel plots mean analysis/router_probe_times_strategy_match_rate for each reconstruction family at beta 1, 0.1, and 0.01. Each point averages the complete single-task datasets available for that family and setting; error bars are standard deviations across those tasks. The two facets split constant beta from the linear-100% beta schedule, and the raw reconstruction family is labeled as the baseline ELBO.

Analogical Consistency

The analogical evaluation reuses a latent sample across paired inputs and scores whether the inferred strategy sets overlap. The table below ranks fixed loss settings by mean analogical_steering/pair_set_overlap_rate across complete single-task datasets with analogical summaries. The May01 and May02 follow-up settings are now included.

The correlation table below compares analogical set overlap with router-side latent metrics, final-answer accuracy, sampled strategy-match rate, and reconstruction metrics in this single-task slice.

Reconstruction Frontier

The log-x reconstruction frontier plots reconstruction loss measures against matched-strategy router score for complete runs. The first panel uses the final baseline-normalized reconstruction loss; the following panels split out distribution-level normalized reconstruction, example-level inverse-surprisal reconstruction, and token-level concentration-corrected reconstruction.

Paired Contrasts

The contrast table reports matched single-task deltas. Positive deltas mean the first setting in the contrast has higher matched-strategy router score.

Dynamics And Run Cards

The dynamics figure aggregates validation curves by reconstruction family. The run-card figure shows selected top and bottom complete runs with router and posterior latent snapshots.

Top complete runs by matched-strategy router score:

Bottom complete runs by matched-strategy router score:

Repo-Side Artifacts

• Aggregate directory: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429
• Report: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/report.md
• Brief: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/brief.yaml
• Findings JSON: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/findings.json
• Notebook: path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/notebook.ipynb