Unified Apr24+Apr26 Reconstruction Stack: Analysis

Back to hub · Results

Agent-drafted interpretation, pending human review

The top of the combined matched-strategy router ranking is effectively a cluster rather than a single clear winner. Apr26 dist_plus_tokcc_a0p5_b0p1_d50_lin50 is numerically first at 0.8556, Apr26 exinv_plus_tokcc_a0p5_b0p01_d50_lin50 is second at 0.8533, and Apr24 dist_plus_tokcc_a0p5_b0p01_d50_lin50 is third at 0.8532. The matched Apr26-vs-Apr24 delta for the dist+cc-TWR delayed-linear continuation is only +0.0024 mean and -0.0112 median, so the beta 0.1 continuation does not look like a decisive improvement over the Apr24 beta 0.01 anchor on the headline metric.

The raw-baseline story changes once low-beta delayed-linear raw reconstruction is included. Apr24 raw beta-1 baselines are all near 0.44 matched score despite near-ceiling final answer accuracy. Apr26 raw_b0p01_const improves to 0.5745, and Apr26 raw_b0p01_d50_lin50 jumps to 0.8429, close to the top normalized rows. The matched contrast for Apr26 low-beta delayed-linear raw versus Apr24 beta-1 delayed-linear raw is +0.4033 mean. This suggests the earlier raw-baseline weakness was not simply "raw reconstruction" in isolation; beta scale and schedule are a major part of the effect.

The strongest non-raw rows still support reconstruction-stack designs that combine broad reconstruction normalization with token or example-level pressure. The best three rows are dist+cc-TWR beta 0.1 delayed-linear, exinv+cc-TWR beta 0.01 delayed-linear, and the Apr24 dist+cc-TWR beta 0.01 delayed-linear anchor. Apr26 exinv+cc-TWR delayed-linear is almost tied with the Apr24 dist+cc-TWR delayed-linear anchor and beats Apr24 example-inverse-surprisal-only delayed-linear by +0.0215 mean. This makes exinv+cc-TWR a credible follow-up candidate, but it does not clearly beat dist+cc-TWR on the current single-seed grid.

cc-TWR alone is more mixed. Apr26 tokcc_a0p25_* rows are surprisingly strong, ranking fourth and fifth overall with matched scores around 0.85, while tokcc_a0p5_* rows rank much lower at 0.81 and 0.78. The alpha 0.25 rows are close to Apr24 dist+cc-TWR anchors by mean, but their baseline-normalized reconstruction ratios are large because that diagnostic is not aligned with the cc-TWR-only objective scale. The safer reading is that cc-TWR-only alpha 0.25 deserves follow-up, while alpha 0.5 has visible task/adaptation failures.

The adaptation-mode figure suggests the best setting is not identical across modes: Apr24 full is led by three_way_equal_b0p01_d50_lin50, Apr24 LoRA by dist_plus_tokcc_a1_b0p01_d50_lin50, Apr24 reinit by dist_plus_tokcc_a0p5_b0p01_const, Apr26 full/LoRA by exinv+cc-TWR delayed-linear, and Apr26 reinit by cc-TWR alpha 0.25 constant. This argues against selecting one objective solely from the global mean if the next run targets a specific adaptation regime.

The reconstruction frontier plots are useful as diagnostics but should not be overread as a single monotone relationship. The Apr24 normalized rows and Apr26 exinv/dist+cc-TWR rows live at low baseline-normalized reconstruction ratios, while Apr26 cc-TWR-only rows have high ratios because the baseline-normalized diagnostic is not their optimized component. The component-loss scatter is therefore a better place to inspect whether a given objective family is improving its own intended reconstruction term.

For next experiments, the most defensible short list is: keep Apr24/Apr26 dist+cc-TWR as the stable anchor, replicate Apr26 exinv+cc-TWR delayed-linear, replicate Apr26 tokcc alpha 0.25, and include Apr26 low-beta delayed-linear raw as a much stronger raw control. Because every collection here uses one seed, the ranking should guide replication rather than settle final claims.

Built with LogoFlowershow