synthetic-pretrained-disentangled-from-entangled-v3 Results

Note

This note reflects the executed notebook at path:experiments/synthetic_sequences/disentangled/analysis/notebooks/14_apr16_ss_from_entangled_v3_analysis.ipynb and the aggregate bundle at path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v3/ after the last non-failed run completed.

Collection State

total planned cells: 30
COMPLETED: 24
FAILED: 6
RUNNING: 0
locally analyzable completed cells: 24

Failed cells in the current snapshot:

sorting_algorithms: beta0p01_const_pv0, beta0p1_lin100_pv0, beta0p05_lin100_pv0, beta0p05_lin100_pv0p01 failed
base_conversion: beta0p02_const_pv0, beta0p01_const_pv0 failed

Current Quantitative Snapshot

From data/final_snapshot.csv:

analyzable completed runs: 24
mean sampled test final-answer accuracy: 0.9915
mean router-sampled probe accuracy: 0.5623
mean posterior-mu probe accuracy: 0.7172
mean baseline-normalized reconstruction loss: 0.6909
mean KL prior loss: 4.0369
mean posterior-variance diagnostic: 568.447
mean router-marginal-KL diagnostic: 5665.769
mean total loss: 1.5494

Per-cell final metrics:

Task	Loss setting	Final accuracy	Router probe	Posterior probe	Norm recon	KL prior	Posterior variance	Total loss
`list_summation`	`beta0p02_const_pv0`	`1.0000`	`0.6017`	`0.7022`	`0.4360`	`0.9809`	`9.86`	`0.4556`
`list_summation`	`beta0p01_const_pv0`	`1.0000`	`0.6427`	`0.7089`	`0.4379`	`1.7045`	`431.80`	`0.4549`
`list_summation`	`beta0p1_lin100_pv0`	`1.0000`	`0.3954`	`0.4214`	`0.9665`	`0.0053`	`409.48`	`0.9670`
`list_summation`	`beta0p05_lin100_pv0`	`1.0000`	`0.3860`	`0.5022`	`0.9658`	`0.0046`	`414.59`	`0.9661`
`list_summation`	`beta0p05_lin100_pv0p01`	`1.0000`	`0.3975`	`0.3813`	`0.9659`	`0.0535`	`372.84`	`4.6969`
`sorting_algorithms`	`beta0p02_const_pv0`	`1.0000`	`0.4335`	`0.6755`	`1.0342`	`2.3481`	`407.66`	`1.0812`
`grid_pathfinding`	`beta0p02_const_pv0`	`1.0000`	`0.6771`	`0.7969`	`0.2732`	`1.0217`	`410.17`	`0.2936`
`grid_pathfinding`	`beta0p01_const_pv0`	`1.0000`	`0.7787`	`0.9933`	`0.0126`	`1.3548`	`4.93`	`0.0262`
`grid_pathfinding`	`beta0p1_lin100_pv0`	`1.0000`	`0.4351`	`0.4669`	`1.0599`	`0.0047`	`452.51`	`1.0603`
`grid_pathfinding`	`beta0p05_lin100_pv0`	`1.0000`	`0.4704`	`0.5427`	`1.0602`	`0.0046`	`881.24`	`1.0605`
`grid_pathfinding`	`beta0p05_lin100_pv0p01`	`1.0000`	`0.4742`	`0.4823`	`1.0627`	`0.0562`	`372.81`	`4.7936`
`linear_equation_solving`	`beta0p02_const_pv0`	`1.0000`	`0.8115`	`1.0000`	`0.0049`	`1.5411`	`2.35`	`0.0357`
`linear_equation_solving`	`beta0p01_const_pv0`	`0.9998`	`0.8987`	`1.0000`	`0.0342`	`1.6298`	`165.42`	`0.0505`
`linear_equation_solving`	`beta0p1_lin100_pv0`	`1.0000`	`0.4340`	`0.8251`	`0.9711`	`0.0129`	`1599.67`	`0.9724`
`linear_equation_solving`	`beta0p05_lin100_pv0`	`1.0000`	`0.4292`	`0.8242`	`0.9715`	`0.0049`	`1987.78`	`0.9718`
`linear_equation_solving`	`beta0p05_lin100_pv0p01`	`1.0000`	`0.4339`	`0.4527`	`0.9717`	`0.0591`	`0.21`	`0.9768`
`base_conversion`	`beta0p1_lin100_pv0`	`1.0000`	`0.7690`	`1.0000`	`0.0288`	`1.4288`	`387.87`	`0.1717`
`base_conversion`	`beta0p05_lin100_pv0`	`1.0000`	`0.8153`	`1.0000`	`0.0434`	`1.4109`	`388.69`	`0.1139`
`base_conversion`	`beta0p05_lin100_pv0p01`	`1.0000`	`0.4375`	`0.4782`	`1.1859`	`0.0433`	`372.80`	`4.9161`
`multidigit_addition`	`beta0p02_const_pv0`	`0.9895`	`0.5076`	`0.7568`	`1.0087`	`5.9748`	`1004.96`	`1.1282`
`multidigit_addition`	`beta0p01_const_pv0`	`1.0000`	`0.8296`	`1.0000`	`0.0227`	`1.4958`	`395.69`	`0.0377`
`multidigit_addition`	`beta0p1_lin100_pv0`	`0.9991`	`0.4545`	`0.8700`	`1.0288`	`25.3448`	`1615.05`	`3.5633`
`multidigit_addition`	`beta0p05_lin100_pv0`	`0.9991`	`0.4391`	`0.6858`	`1.0268`	`13.3288`	`1170.65`	`1.6933`
`multidigit_addition`	`beta0p05_lin100_pv0p01`	`0.8077`	`0.5423`	`0.6469`	`1.0077`	`37.0726`	`383.67`	`6.6981`

Figures

Collection status by task and loss setting. Caption: Collection status over the 6 x 5 task-by-loss-setting design. The current snapshot contains 24 completed cells and 6 failed cells.

Final metric heatmaps by task and loss setting. Caption: Final-state heatmaps for router probe, posterior probe, baseline-normalized reconstruction, and KL prior over the 24 completed cells.

Alignment dynamics by task and loss setting. Caption: Validation final-answer accuracy, router-sampled probe accuracy, and posterior-mu probe accuracy over training.

Objective-term dynamics by task and loss setting. Caption: Baseline-normalized reconstruction, KL prior, and total objective over training.

Auxiliary diagnostic dynamics by task and loss setting. Caption: Posterior-variance and router-marginal-KL diagnostics over training.

Final normalized reconstruction versus router probe. Caption: Final baseline-normalized reconstruction loss versus final router-sampled probe accuracy across the completed cells.

Final router probe versus posterior probe. Caption: Final router-sampled probe accuracy versus final posterior-mu probe accuracy across the completed cells.

Delta versus v1 heatmaps. Caption: Per-task deltas relative to synthetic-pretrained-disentangled-from-entangled-v1 for router probe, posterior probe, and baseline-normalized reconstruction, using the completed cells available in the current snapshot.

synthetic-pretrained-disentangled-from-entangled-v3 Results

Collection State

Current Quantitative Snapshot

Figures

Links to this page: