synthetic-pretrained-disentangled-from-entangled-v5-twr Results

Note

This is an early partial-collection snapshot based on path:experiments/synthetic_sequences/disentangled/analysis/notebooks/16_apr18_ss_from_entangled_v5_twr_analysis.ipynb and path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v5-twr/.

TWR is implemented incorrectly

Token-weighted reconstruction loss is implemented incorrectly here due to a bug in the alignment of $z$ -free tokenized sequence with the $z$ -included tokenized sequence. This unintended behavior stems from the use of the pre-trained model's tokenizer rather than a task-specific tokenizer.

Collection State

total planned cells: 24
COMPLETED_READY: 7
RUNNING: 1
PENDING: 16
locally analyzable completed cells: 7

Cells not included in the current quantitative snapshot:

list_summation: beta0p005_const_pv0_twr0p25 is still running, beta0p005_const_pv0_twr0p5 is still pending
sorting_algorithms: all 4 cells are still pending
grid_pathfinding: beta0p005_const_pv0_twr0p25 is still pending
linear_equation_solving: all 4 cells are still pending
base_conversion: all 4 cells are still pending
multidigit_addition: beta0p005_const_pv0_twr0p25 and beta0p01_lin100_pv0_twr0p5 are still pending

Current Quantitative Snapshot

From data/final_snapshot.csv:

analyzable completed runs: 7
mean sampled test final-answer accuracy: 0.9893
mean router-sampled probe accuracy: 0.6074
mean posterior-mu probe accuracy: 0.8087
mean baseline-normalized reconstruction loss: 0.5921
mean token-weighted reconstruction loss: 0.000016
mean KL prior loss: 2.4572
mean total loss: 0.6142

Per-cell final metrics:

Task	Loss setting	Final accuracy	Router probe	Posterior probe	Norm recon	Token-weighted recon	KL prior	Total loss
`list_summation`	`beta0p01_lin100_pv0_twr0p25`	`1.0000`	`0.8070`	`0.9920`	`0.0306`	`0.0000`	`1.4653`	`0.0452`
`list_summation`	`beta0p01_lin100_pv0_twr0p5`	`1.0000`	`0.3880`	`0.5432`	`0.9657`	`0.0001`	`0.0083`	`0.9659`
`grid_pathfinding`	`beta0p005_const_pv0_twr0p5`	`1.0000`	`0.7780`	`1.0000`	`0.0111`	`0.0000`	`1.4126`	`0.0181`
`grid_pathfinding`	`beta0p01_lin100_pv0_twr0p25`	`1.0000`	`0.4233`	`0.8066`	`1.0592`	`0.0000`	`0.0002`	`1.0592`
`grid_pathfinding`	`beta0p01_lin100_pv0_twr0p5`	`1.0000`	`0.4640`	`0.6236`	`1.0604`	`0.0000`	`0.0044`	`1.0604`
`multidigit_addition`	`beta0p005_const_pv0_twr0p5`	`0.9991`	`0.8768`	`1.0000`	`0.0067`	`0.0000`	`2.1843`	`0.0176`
`multidigit_addition`	`beta0p01_lin100_pv0_twr0p25`	`0.9260`	`0.5143`	`0.6956`	`1.0113`	`0.0000`	`12.1253`	`1.1326`

Observations By Design Question

Incremental effect relative to the `v4` no-TWR baselines

The current notebook writes a matched comparison against the corresponding v4 no-TWR settings. Among the 7 completed cells:

list_summation / beta0p01_lin100_pv0_twr0p25 is above the matched v4 baseline on router probe (+0.1643) and posterior probe (+0.2831), with lower normalized reconstruction (-0.4073)
list_summation / beta0p01_lin100_pv0_twr0p5 is below the matched v4 baseline on router probe (-0.2546) and posterior probe (-0.1657), with higher normalized reconstruction (+0.5278)
grid_pathfinding / beta0p005_const_pv0_twr0p5 is above the matched v4 baseline on router probe (+0.3288) and posterior probe (+0.5170), with lower normalized reconstruction (-1.0481)
grid_pathfinding / beta0p01_lin100_pv0_twr0p25 and ...twr0p5 are both below the matched v4 warmup baseline on router probe and posterior probe, with higher normalized reconstruction
multidigit_addition / beta0p005_const_pv0_twr0p5 is above the matched v4 constant baseline on router probe (+0.0932) with unchanged posterior probe (+0.0000)
multidigit_addition / beta0p01_lin100_pv0_twr0p25 is below the matched v4 warmup baseline on router probe (-0.3153) and posterior probe (-0.3044), with higher normalized reconstruction (+0.9886)

`token_weighted_reconstruction_weight=0.25` versus `0.5`

The only currently completed matched 0.25 versus 0.5 comparisons are the beta=0.01 full-warmup cells on list_summation and grid_pathfinding.

on list_summation, the 0.25 setting is higher on router probe (0.8070 vs 0.3880) and posterior probe (0.9920 vs 0.5432), with lower normalized reconstruction (0.0306 vs 0.9657)
on grid_pathfinding, the 0.5 setting is slightly higher on router probe (0.4640 vs 0.4233), while the 0.25 setting is higher on posterior probe (0.8066 vs 0.6236)

No completed 0.25 versus 0.5 comparison is available yet in the beta=0.005 constant regime.

Magnitude of the logged token-weighted reconstruction term

Across the 7 completed cells, the logged loss/val/token_weighted_reconstruction_loss values are numerically very small in the current snapshot. The per-cell table rounds them to 0.0000 except for list_summation / beta0p01_lin100_pv0_twr0p5, which rounds to 0.0001. The current mean over the completed cells is 0.000016.

Figures

Collection status by task and loss setting. Caption: Current partial-collection status over the 6 x 4 task-by-loss-setting design. The current snapshot contains 7 completed cells, 1 running cell, and 16 pending cells.

Final metric heatmaps by task and loss setting. Caption: Final-state heatmaps for router probe, posterior probe, baseline-normalized reconstruction, and KL prior over the 7 completed cells.

Alignment dynamics by task and loss setting. Caption: Validation final-answer accuracy, router-sampled probe accuracy, and posterior-mu probe accuracy over training.

Objective-term dynamics by task and loss setting. Caption: Baseline-normalized reconstruction, token-weighted reconstruction, KL prior, and total objective over training.

Auxiliary diagnostic dynamics by task and loss setting. Caption: Posterior-variance and router-marginal-KL diagnostics over training.

Final normalized reconstruction versus router probe. Caption: Final baseline-normalized reconstruction loss versus final router-sampled probe accuracy across the completed cells.

Final router probe versus posterior probe. Caption: Final router-sampled probe accuracy versus final posterior-mu probe accuracy across the completed cells.

Delta versus matched v4 no-TWR baselines. Caption: Per-task deltas relative to the matched v4 no-TWR baselines for router probe, posterior probe, and baseline-normalized reconstruction, using the 7 completed cells available in the current snapshot.