Pythia-410M Diagnostic
Pythia-410M Diagnostic
Back to usable SmolLM2 results
Pythia-410M did not adequately learn the multi-task source benchmark. The one-epoch entangled source reached final test strategy match 0.6051; the fresh three-epoch source improved to 0.8376, but remained below the operational 0.95 source-learning prerequisite and far below SmolLM2-360M (0.9937).
The downstream Pythia factorization endpoints are therefore confounded by an inadequate source model and are not usable as evidence about factorization quality. They remain recorded only as diagnostics. Five factorizations of the one-epoch source completed; token inverse failed after numerical instability produced invalid sampling probabilities.
One-Epoch Source and Factorizations
The entangled SFT job 2163505 completed in 1:40:51. It ended with train cross-entropy 0.2319, validation cross-entropy 0.2339, test strategy match 0.6051, and sampled-validation strategy match 0.6113.
| Role | Job | State | Distributional Fidelity | Strategy Decodability | Analogical Transfer |
|---|---|---|---|---|---|
| Standard CVAE | 2163507 | completed | 0.6699 | 0.7197 | 0.2012 |
| β-ELBO | 2163508 | completed | 0.7284 | 0.5880 | 0.2246 |
| Global normalization | 2163509 | completed | 0.8277 | 0.8205 | 0.6357 |
| Token weighting | 2163510 | completed | 0.7672 | 0.5591 | 0.2188 |
| Global + token | 2163511 | completed | 0.7768 | 0.8291 | 0.5283 |
| Token inverse | 2163512 | failed | — | — | — |
Three-Epoch Controlled Follow-Up
The three-epoch source started fresh from EleutherAI/pythia-410m. The only intended training change was num_epochs: 1 -> 3; the seed, data, maximum sequences per epoch, LoRA rank and targets, optimizer, batch sizes, learning-rate schedule family, and adaptation mode remained fixed. The schedule spanned all three epochs.
The source job 2163970 completed in 4:17:15. Five of six dependent factorization jobs completed; token inverse failed.
| Role | Job | State | Distributional Fidelity | Strategy Decodability | Analogical Transfer |
|---|---|---|---|---|---|
| Standard CVAE | 2163971 | completed | 0.8444 | 0.6803 | 0.2861 |
| β-ELBO | 2163972 | completed | 0.8813 | 0.6003 | 0.3066 |
| Global normalization | 2163973 | completed | 0.8927 | 0.6292 | 0.3105 |
| Token weighting | 2163974 | completed | 0.8820 | 0.5941 | 0.2900 |
| Global + token | 2163975 | completed | 0.9241 | 0.7131 | 0.3896 |
| Token inverse | 2163976 | failed | — | — | — |
Distributional Fidelity is aggregate final-test strategy match. Strategy Decodability is the equal-weight mean of the six task-conditioned router-sampled latent linear-probe accuracies. Analogical Transfer is final aggregate pair-set overlap over 1,024 paired evaluations.
The complete task-level table is stored at path:experiments/synthetic_sequences/disentangled/analysis/collections/pythia-410m-3epoch-followup_20260725/data/per_task_metrics.csv. The reproducible report and machine-readable findings remain under path:experiments/synthetic_sequences/disentangled/analysis/collections/pythia-410m-3epoch-followup_20260725/.
