from-entangled-v2 continuous-v1 Results

Note

This report reflects the completed 36/36 collection based on path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-continuous-v1/. Earlier synthetic-pretrained-disentangled-from-entangled-v{X} collections were used only for figure-layout inspiration in the repo-side analysis. No direct numerical comparison to those collections is made here.

Collection State

total planned cells: 36
COMPLETED_READY: 36
RUNNING: 0
locally analyzable completed cells: 36
cells excluded from the quantitative snapshot: none

Current Quantitative Snapshot

From data/final_snapshot.csv:

analyzable completed runs: 36
mean sampled test final-answer accuracy: 0.9980
mean router-sampled probe accuracy: 0.6663
mean posterior-mu probe accuracy: 0.8274
mean baseline-normalized reconstruction loss: 0.2327
mean strategy-match rate: 0.9958
mean unique-strategy rate: 0.9292

Per-loss-setting means:

Loss setting	Final accuracy	Router probe	Posterior probe	Norm recon	Strategy match	Unique strategy
`beta1_const_pv0_unnorm`	`0.9979`	`0.4426`	`0.5540`	`0.0379`	`0.9907`	`0.9164`
`beta0_const_pv0`	`0.9900`	`0.8940`	`1.0000`	`0.0000`	`0.9845`	`0.9474`
`beta0p005_const_pv0`	`1.0000`	`0.6165`	`0.7470`	`0.5836`	`0.9999`	`0.9255`
`beta0p01_lin100_pv0`	`1.0000`	`0.6573`	`0.8283`	`0.3421`	`1.0000`	`0.9280`
`beta0p005_const_pv0_twr0p5`	`0.9999`	`0.7315`	`0.9160`	`0.1819`	`0.9999`	`0.9296`
`beta0p01_lin100_pv0_twr0p5`	`1.0000`	`0.6938`	`0.9191`	`0.2506`	`0.9997`	`0.9282`

Top And Bottom Runs By Router Probe

Top runs:

Task	Loss setting	Final accuracy	Router probe	Posterior probe	Norm recon	Strategy match
`sorting_algorithms`	`beta0_const_pv0`	`0.9911`	`0.9263`	`1.0000`	`0.0000`	`0.9657`
`base_conversion`	`beta0p005_const_pv0`	`1.0000`	`0.9248`	`1.0000`	`0.0163`	`1.0000`
`grid_pathfinding`	`beta0_const_pv0`	`1.0000`	`0.9037`	`1.0000`	`0.0000`	`1.0000`
`list_summation`	`beta0_const_pv0`	`0.9886`	`0.8945`	`1.0000`	`0.0000`	`0.9883`
`base_conversion`	`beta0_const_pv0`	`0.9789`	`0.8766`	`1.0000`	`0.0000`	`0.9735`
`multidigit_addition`	`beta0p005_const_pv0`	`1.0000`	`0.8704`	`1.0000`	`0.0155`	`1.0000`

Bottom runs:

Task	Loss setting	Final accuracy	Router probe	Posterior probe	Norm recon	Strategy match
`sorting_algorithms`	`beta1_const_pv0_unnorm`	`0.9996`	`0.3879`	`0.5352`	`0.0300`	`0.9570`
`sorting_algorithms`	`beta0p01_lin100_pv0_twr0p5`	`1.0000`	`0.3988`	`0.6094`	`1.0616`	`0.9986`
`list_summation`	`beta0p005_const_pv0`	`1.0000`	`0.4022`	`0.3997`	`0.9652`	`1.0000`
`list_summation`	`beta0p01_lin100_pv0`	`1.0000`	`0.4091`	`0.5099`	`0.9646`	`1.0000`
`sorting_algorithms`	`beta0p005_const_pv0_twr0p5`	`1.0000`	`0.4103`	`0.5199`	`1.0478`	`0.9998`
`list_summation`	`beta1_const_pv0_unnorm`	`0.9999`	`0.4176`	`0.4797`	`0.0425`	`0.9999`

Observations By Design Question

ELBO anchor versus low-KL settings

Across the full 36 completed cells, the beta1_const_pv0_unnorm ELBO anchor has the lowest mean router probe (0.4426) and the lowest mean posterior probe (0.5540) of the six settings. The completed beta0_const_pv0 rows have the highest mean router probe (0.8940) and posterior probe (1.0000), with baseline-normalized reconstruction values at or near zero on all six tasks.

Constant `beta=0.005` versus linear-warmup `beta=0.01`

The completed non-TWR rows show higher mean router probe and posterior probe for beta0p01_lin100_pv0 (0.6573, 0.8283) than for beta0p005_const_pv0 (0.6165, 0.7470). The corresponding mean baseline-normalized reconstruction values are 0.3421 for beta0p01_lin100_pv0 and 0.5836 for beta0p005_const_pv0.

Matched TWR versus no-TWR within this collection

The matched constant-beta comparisons are now available for all six tasks. Their mean deltas are +0.1150 router probe, +0.1690 posterior probe, -0.0000 final accuracy, and -0.4016 normalized reconstruction for beta0p005_const_pv0_twr0p5 relative to beta0p005_const_pv0.

The matched linear-warmup comparisons are also available for all six tasks. Their mean deltas are +0.0365 router probe, +0.0909 posterior probe, -0.0000 final accuracy, and -0.0915 normalized reconstruction for beta0p01_lin100_pv0_twr0p5 relative to beta0p01_lin100_pv0.

Task-level matched TWR deltas show large positive constant-beta gains on list_summation, grid_pathfinding, and linear_equation_solving, together with negative constant-beta router-probe deltas on sorting_algorithms, base_conversion, and multidigit_addition. The linear-warmup TWR deltas are smaller in magnitude overall: list_summation is the largest positive router-probe case, multidigit_addition is modestly positive, and sorting_algorithms, grid_pathfinding, linear_equation_solving, and base_conversion stay near zero or negative.

Task heterogeneity

sorting_algorithms has the lowest unique-strategy rates across the full collection, ranging from 0.8469 to 0.8842 across the six settings. grid_pathfinding reaches both strategy-match rate 1.0000 and final-answer accuracy 1.0000 under all six settings. linear_equation_solving, base_conversion, and multidigit_addition each have four settings with both values at 1.0000.

Loss-Setting Deep Dives

Note

The strongest setting by mean final router-sampled probe accuracy is beta0_const_pv0, and the second-best setting by the same metric is beta0p005_const_pv0_twr0p5. Their within-setting analyses live in b0-const-deepdive-results and b0p005-const-twr0p5-deepdive-results. The collection-level figure gallery below stays focused on cross-setting comparisons.

Figures

Collection status by task and loss setting. Caption: Completed collection status over the 6 x 6 task-by-loss-setting design. All 36 cells are completed and included in the local quantitative snapshot.

Final metric heatmaps by task and loss setting. Caption: Completed-cell heatmaps for final answer accuracy, router probe, posterior probe, strategy-match rate, unique-strategy rate, and baseline-normalized reconstruction.

Router probe summary by loss setting. Caption: Mean final router-probe accuracy for each loss setting, aggregated over all six tasks. Bars show the across-task mean and the overlaid markers show the task-level values contributing to each mean.

Missing multidigit_addition / b0 const router-probe cell

In the aggregated metrics, sampled_test_final/router_sampled_z/linear_probe_accuracy for this run is NaN. The validation probe is present at earlier checkpoints, including 0.9968 near 99.16% training. It's unclear why the final checkpoint fails to compute this metric. Can look into it later. Validation gives us a view of what it looks like.

Alignment dynamics by task and loss setting. Caption: Validation final-answer accuracy, router-sampled probe accuracy, and posterior-mu probe accuracy over training.

Objective dynamics by task and loss setting. Caption: Baseline-normalized reconstruction, token-weighted reconstruction, KL prior, and total objective over training.

Final normalized reconstruction versus router probe. Caption: Final baseline-normalized reconstruction loss versus final router-sampled probe accuracy across the completed collection.

Baseline-normalized reconstruction loss in unnormed is incorrect

When the --baseline-normalized-reconstruction flag is False (e.g., in the b1 const unnorm loss setting), the current train_disentangled.py script computes loss/val/baseline_normalized_reconstruction_loss = loss/val/reconstruction_loss. So, the lower left quadrant is incorrect. The baseline-normalized reconstruction loss appear low (near 0) for b1 const unnorm, but this is actually just the reconstruction loss and the normalized reconstruction would likely be near 1.

Final router probe versus posterior probe. Caption: Final router-sampled probe accuracy versus final posterior-mu probe accuracy across the completed collection.

Matched with-TWR versus without-TWR final metrics within the current collection. Caption: Within-collection matched bars for router probe, posterior probe, final accuracy, and normalized reconstruction. Each task shows the paired no-TWR and TWR values for the constant-beta=0.005 and linear-warmup beta=0.01 settings.

Matched router-probe dynamics for no-TWR versus +TWR settings. Caption: Validation router-probe accuracy over training for the matched constant-beta=0.005 and linear-warmup beta=0.01 comparisons. Rows are tasks, columns are base-loss families, and each panel overlays the no-TWR and +TWR trajectories across the full completed collection.