synthetic-pretrained-disentangled-from-entangled-v4 Results
synthetic-pretrained-disentangled-from-entangled-v4 Results
This is a partial-collection snapshot based on path:experiments/synthetic_sequences/disentangled/analysis/notebooks/15_apr17_ss_from_entangled_v4_analysis.ipynb and path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v4/.
Collection State
- total planned cells:
24 COMPLETED_READY:19FAILED:3RUNNING:2- locally analyzable completed cells:
19
Cells not included in the current quantitative snapshot:
sorting_algorithms:beta0_const_pv0,beta0p005_const_pv0,beta0p01_lin100_pv0failedbase_conversion:beta0_const_pv0,beta0p005_const_pv0still running
Current Quantitative Snapshot
From data/final_snapshot.csv:
- analyzable completed runs:
19 - mean sampled test final-answer accuracy:
0.9825 - mean router-sampled probe accuracy:
0.6793 - mean posterior-mu probe accuracy:
0.8312 - mean baseline-normalized reconstruction loss:
0.3689 - mean KL prior loss:
38747.7080 - mean posterior-variance diagnostic:
416.625 - mean router-marginal-KL diagnostic:
6521.506 - mean total loss:
0.6903
Per-cell final metrics:
| Task | Loss setting | Final accuracy | Router probe | Posterior probe | Norm recon | KL prior | Posterior variance | Total loss |
|---|---|---|---|---|---|---|---|---|
list_summation | beta0_const_pv0 | 0.9991 | 0.7147 | 1.0000 | 0.0000 | 588906.7160 | 0.02 | 0.0000 |
list_summation | beta0p005_const_pv0 | 1.0000 | 0.3851 | 0.4724 | 0.9657 | 1.1220 | 10.54 | 0.9713 |
list_summation | beta0p01_lin100_pv0 | 1.0000 | 0.6427 | 0.7089 | 0.4379 | 1.7045 | 431.80 | 0.4549 |
list_summation | beta0p01_lin100_pv0p005 | 1.0000 | 0.3936 | 0.4178 | 0.9659 | 0.0827 | 0.12 | 0.9673 |
sorting_algorithms | beta0p01_lin100_pv0p005 | 0.9886 | 0.4237 | 0.5943 | 1.0496 | 15.5839 | 377.07 | 3.0908 |
grid_pathfinding | beta0_const_pv0 | 1.0000 | 0.9214 | 1.0000 | 0.0000 | 11010.7349 | 0.01 | 0.0000 |
grid_pathfinding | beta0p005_const_pv0 | 0.9998 | 0.4492 | 0.4830 | 1.0591 | 4.8405 | 478.46 | 1.0833 |
grid_pathfinding | beta0p01_lin100_pv0 | 1.0000 | 0.7758 | 1.0000 | 0.0060 | 1.3454 | 3.09 | 0.0195 |
grid_pathfinding | beta0p01_lin100_pv0p005 | 1.0000 | 0.4514 | 0.5060 | 1.0587 | 0.1085 | 372.74 | 2.9234 |
linear_equation_solving | beta0_const_pv0 | 0.9895 | 0.9302 | 1.0000 | 0.0000 | 15906.1288 | 0.01 | 0.0000 |
linear_equation_solving | beta0p005_const_pv0 | 1.0000 | 0.8014 | 1.0000 | 0.0012 | 1.6891 | 8.58 | 0.0096 |
linear_equation_solving | beta0p01_lin100_pv0 | 1.0000 | 0.5883 | 0.9563 | 0.3843 | 3.4374 | 2980.96 | 0.4186 |
linear_equation_solving | beta0p01_lin100_pv0p005 | 1.0000 | 0.7611 | 0.9639 | 0.0071 | 1.6106 | 0.26 | 0.0245 |
base_conversion | beta0p01_lin100_pv0 | 1.0000 | 0.8092 | 1.0000 | 0.0062 | 2.1925 | 2079.81 | 0.0281 |
base_conversion | beta0p01_lin100_pv0p005 | 0.9999 | 0.8189 | 0.9624 | 0.0181 | 1.5503 | 0.20 | 0.0346 |
multidigit_addition | beta0_const_pv0 | 0.8017 | 0.8987 | 1.0000 | 0.0000 | 120331.2080 | 0.05 | 0.0000 |
multidigit_addition | beta0p005_const_pv0 | 1.0000 | 0.7836 | 1.0000 | 0.0192 | 1.5957 | 399.81 | 0.0272 |
multidigit_addition | beta0p01_lin100_pv0 | 1.0000 | 0.8296 | 1.0000 | 0.0227 | 1.4958 | 395.69 | 0.0377 |
multidigit_addition | beta0p01_lin100_pv0p005 | 0.8888 | 0.5289 | 0.7269 | 1.0078 | 13.3063 | 376.67 | 3.0243 |
Observations By Design Question
beta=0 versus beta=0.005
Among the currently completed task-setting cells, the zero-beta setting completed for list_summation, grid_pathfinding, linear_equation_solving, and multidigit_addition. On those tasks, the zero-beta runs show near-zero normalized reconstruction loss and much larger KL-prior values than the beta=0.005 runs. Router probe is higher under beta=0 on all four currently comparable tasks, while final-answer accuracy is lower for linear_equation_solving and multidigit_addition.
beta=0.005 constant versus beta=0.01 with full warmup
This comparison is currently available on list_summation, grid_pathfinding, linear_equation_solving, and multidigit_addition, with base_conversion still incomplete. The completed cells show mixed behavior: beta=0.01 full warmup is stronger on router probe for list_summation, grid_pathfinding, and multidigit_addition, while beta=0.005 constant remains stronger on linear_equation_solving.
Matched mild-PV comparison at beta=0.01
The matched full-warmup beta=0.01 comparison is already available on every task except the three failed sorting_algorithms non-PV cells. The pv=0.005 setting strongly suppresses posterior variance relative to beta0p01_lin100_pv0, but the completed cells show task-dependent changes in router probe, posterior probe, and accuracy rather than a single directional pattern.
Figures
Caption: Current partial-collection status over the 6 x 4 task-by-loss-setting design. The current snapshot contains 19 completed cells, 3 failed cells, and 2 running cells.
Caption: Final-state heatmaps for router probe, posterior probe, baseline-normalized reconstruction, and KL prior over the 19 completed cells.
Caption: Validation final-answer accuracy, router-sampled probe accuracy, and posterior-mu probe accuracy over training.
Caption: Baseline-normalized reconstruction, KL prior, and total objective over training.
Caption: Posterior-variance and router-marginal-KL diagnostics over training.
Caption: Final baseline-normalized reconstruction loss versus final router-sampled probe accuracy across the completed cells.
Caption: Final router-sampled probe accuracy versus final posterior-mu probe accuracy across the completed cells.
Caption: Per-task deltas relative to synthetic-pretrained-disentangled-from-entangled-v1 for router probe, posterior probe, and baseline-normalized reconstruction, using the completed cells available in the current snapshot.
