Full vs Reinit Dynamics Deep-Dive Results
Full vs Reinit Dynamics Deep-Dive Results
This note isolates the adaptation-mode comparison. The trajectory figure focuses on the three settings that matter most for the current follow-up discussion: beta0p01_d50_lin50, beta0p01_cyclin25_w12p5, and beta0p1_d50_lin50. The heatmap then adds the full six-setting final-gap view.
Snapshot
- focused settings in the trajectory figure:
beta0p01_d50_lin50,beta0p01_cyclin25_w12p5,beta0p1_d50_lin50 - shared early pattern: at the first eval checkpoint,
fullvalidation accuracy is already near0.97, whilereinitstarts around0.14to0.15for all three selected settings - shared early pattern: the first-checkpoint router probe goes the other way, with
reinitaround0.91to0.94andfullaround0.42to0.44
At the final shared validation checkpoint, the remaining mode gaps are:
| loss_setting | delta_full_minus_reinit_val_accuracy | delta_full_minus_reinit_val_router_probe | delta_full_minus_reinit_val_norm_recon | ratio_reinit_to_full_val_norm_recon |
|---|---|---|---|---|
| beta0p01_d50_lin50 | 0.1784 | 0.0161 | -0.0029 | 1.0629 |
| beta0p01_cyclin25_w12p5 | 0.0816 | -0.0063 | 0.0015 | 0.9747 |
| beta0p1_d50_lin50 | 0.0765 | -0.0291 | -0.0186 | 1.3024 |
Selected-Setting Trajectories
Across the selected settings, the main persistent full-versus-reinit difference is validation accuracy and reconstruction, not the final router-probe level. beta0p01_d50_lin50 keeps the largest selected-setting final accuracy gap, while beta0p01_cyclin25_w12p5 closes the mode gap more aggressively through the second half of training.
Caption: Mean validation accuracy, router probe, strategy match rate, matched-strategy router score, normalized reconstruction, and KL over training for the three selected settings, comparing full directly against reinit.
Final Gap Structure Across All Settings
The all-setting heatmap shows that the most severe full-minus-reinit final accuracy gaps are concentrated in linear_equation_solving, base_conversion, and multidigit_addition. The strongest collapse cluster remains beta0_unnorm, but beta0p01_d50_lin50 still leaves large positive full-minus-reinit accuracy gaps on the same three tasks. By contrast, beta1_d50_lin50 and beta0p1_cyclin25_w12p5 are much closer to mode-invariant on final accuracy, even though their router gaps can still move slightly negative or positive by task.
Caption: Task-by-setting heatmaps for final full-minus-reinit gaps in final accuracy, router probe, and router probe x final answer accuracy.
Latent-Space Structure Over Training
The latent-geometry comparison below anchors on beta0p1_d50_lin50, the strongest overall setting by the current router probe x final answer accuracy ranking. The full and reinit views are split into separate figures so each mode gets more panel space. Each panel uses a fixed PCA basis per task-and-mode cell across the full saved frame stream. Router-sampled panels use continuous_val_router_sampled_z_frames.jsonl; posterior-side panels use the available posterior-mean stream, continuous_val_posterior_mu_frames.jsonl, because this collection does not contain posterior-sampled-z frame artifacts.
Caption: Router-sampled latent geometry over training for beta0p1_d50_lin50 under full adaptation.
Caption: Selected router-sampled latent checkpoints for beta0p1_d50_lin50 under full adaptation.
Caption: Posterior-mu latent geometry over training for beta0p1_d50_lin50 under full adaptation.
Caption: Smoothed posterior-mu latent geometry for beta0p1_d50_lin50 under full adaptation, using linear interpolation of projected point positions between saved eval frames.
Caption: Early-emphasis posterior-mu latent checkpoints for beta0p1_d50_lin50 under full adaptation: approximately 2%, 6%, 10%, 20%, 50%, and 100% training progress.
Caption: Router-sampled latent geometry over training for beta0p1_d50_lin50 under reinit adaptation.
Caption: Selected router-sampled latent checkpoints for beta0p1_d50_lin50 under reinit adaptation.
Caption: Posterior-mu latent geometry over training for beta0p1_d50_lin50 under reinit adaptation.
Caption: Smoothed posterior-mu latent geometry for beta0p1_d50_lin50 under reinit adaptation, using linear interpolation of projected point positions between saved eval frames.
Caption: Early-emphasis posterior-mu latent checkpoints for beta0p1_d50_lin50 under reinit adaptation: approximately 2%, 6%, 10%, 20%, 50%, and 100% training progress.
