Additional Model Families Results

Back to design

Completeness and Source Learning

All six entangled SFT jobs completed. Every non-Pythia source passed the 0.95 source-learning prerequisite:

At 2026-07-27 15:16 EDT, the full logical-job graph contained 42 jobs: 40 completed, 2 failed, 0 running, and 0 pending. The two failed logical jobs are the Pythia Token inverse arms.

The six original Gemma jobs with evaluation batch size 128 exhausted H100 memory during scheduled validation in the frozen-reference token-level cross-entropy path. Their batch-size-16 retries 2171730–2171735 all completed with exit code 0.

Post-run inspection found that the final continuous-generation evaluator reset the same random seed for every evaluation chunk. Commit 5bb11913 assigned a deterministic distinct seed to each chunk, and the corrected retries were pinned to that commit. Nevertheless, the new Gemma runs still collapsed between scheduled validation and the large final-test generation pass: final-test parse success was 0.1178–0.2897 for Gemma 270M and 0.3054–0.3549 for Gemma 1B, with strategy-match rate at most 0.0024. The seed reuse was therefore not the root cause of the Gemma endpoint discontinuity, and those final-test fidelity values remain invalid for scientific comparison.

Completed Factorization Endpoints

SmolLM2-360M

OLMo 2 1B

Gemma validation and analogical evidence

The table below reports all twelve completed Gemma factorizations. Fidelity and decodability are taken from the last scheduled validation at training step 1563; Analogical Transfer is the final-checkpoint pair_set_overlap_rate. These are recoverable results, but validation fidelity is not a matched replacement for the paper's final-test fidelity.

Bars are aggregate rates and open points are the six benchmark tasks. The first two columns use the last scheduled validation; the third uses the final analogical evaluation. The figure intentionally excludes invalid large-final-test fidelity.

Distributional Fidelity is final aggregate test strategy-match rate. Strategy Alignment follows the paper and is final Analogical Transfer (pair_set_overlap_rate). Router Strategy Decodability is the mean of the six task-level linear-probe accuracies, ignoring a missing task value in Gemma 270M Token inverse.

Bars show aggregate endpoints and open points show the six benchmark tasks. The SmolLM2 and OLMo panels are current. The Gemma panels preserve the earlier invalid final-test diagnostic and must not be read as model comparisons; use the dedicated Gemma validation figure above.

The Router Strategy Decodability figure has the same qualification for Gemma. Its current scheduled-validation values are reported in the dedicated figure and table above.

Historical Runtime Context

The critical path assumes downstream factorizations run concurrently. SmolLM2 uses the six exact selected roles and includes their final analogical evaluation; the Qwen rows summarize the eight-job historical paper-method grids, so the comparison is operational rather than perfectly matched.

Task-level endpoints and runtime records are preserved in path:experiments/synthetic_sequences/disentangled/analysis/collections/cross-family-small_20260724/data/. Pythia observations are separated in pythia because its source did not learn the benchmark adequately.

OLMo 2 1B SFT completed in 6:39:05; its six factorization jobs took 4:53:23–5:44:27. Gemma 3 270M SFT completed in 2:56:14; its six successful factorization jobs took 4:29:53–4:54:23. Gemma 3 1B PT SFT completed in 5:36:13; its six successful factorization jobs took 6:31:56–7:04:52.

The reusable endpoint tables are at path:experiments/synthetic_sequences/disentangled/analysis/collections/cross-family-small_20260724/data/additional_model_endpoints.csv and path:experiments/synthetic_sequences/disentangled/analysis/collections/cross-family-small_20260724/data/additional_model_task_endpoints.csv.

The Gemma validation-specific tables are at path:experiments/synthetic_sequences/disentangled/analysis/collections/cross-family-small_20260724/data/gemma_validation_endpoints.csv and path:experiments/synthetic_sequences/disentangled/analysis/collections/cross-family-small_20260724/data/gemma_validation_task_endpoints.csv.