Sum Token Aggregation Ablation: Analysis
Sum Token Aggregation Ablation: Analysis
The first-pass aggregate suggests that sum-based token aggregation is not a favorable replacement for sequence-mean aggregation in this compact Qwen2.5 0.5B multi-task setting. The common-cell matched-router deltas are negative, while final answer accuracy remains saturated. A scale-controlled follow-up would be needed before treating this as evidence about token aggregation independent of objective magnitude.
What The Local Data Can Answer
| diagnostic | local_data | fetch_needed | local_source | missing_data_note |
|---|---|---|---|---|
| KL-to-reconstruction trade-off | yes | no | summary.csv | |
| KL/reconstruction ratio | yes | no | summary.csv | |
| Sum scale vs mean scale | yes | no | summary.csv + final_05b_summary.csv | |
| Mean-vs-sum objective scale | yes | no | mean_vs_sum_common_cells.csv + final_05b_summary.csv | |
| Posterior-router gap | yes | no | summary.csv | |
| KL dynamics over training | yes | no | sum_eval_history_long.csv | |
| Reconstruction dynamics | yes | no | sum_eval_history_long.csv | |
| Initialization / first-eval scale | partial | yes | sum_eval_history_long.csv | earliest sum validation checkpoint is step 31; exact initialization metrics are absent |
| Token-count dominance | no | yes | requires raw records or a new per-task/per-example aggregate | |
| Fairness of sum collection | yes | no | summary.csv + slurmkit collection YAML |
Fairness Assessment
| claim | status | evidence |
|---|---|---|
| planned sum grid completeness | supported | 12/12 sum cells have final metrics |
| narrow fixed setting coverage | supported | fixed Qwen2.5 0.5B, multi_task, z64 continuous, LoRA, one seed, token aggregation sum |
| common-cell mean comparison | partial | 8 common cells; beta=1 sum rows lack matching mean-reference rows |
| aggregation-only causal attribution | not supported | sum rows use constant beta, while matched mean-reference rows use lin100 schedules |
| training-dynamics diagnosis | supported locally | sum_eval_history_long.csv has 12 runs x 51 validation checkpoints, plus per-task validation traces |
The collection is a fair representation of the narrow sum aggregation mode grid that was launched: it covers four reconstruction families, three beta settings, one model, one task mixture, one latent dimension, and one seed with all cells complete. It is not a fair standalone representation of sum aggregation in general, because it lacks seed replication, model-size replication, matched constant-beta sequence-mean rows, beta 1.0 sequence-mean matches, and local sum training histories.
Diagnostic Questions
- Is sum failing because raw per-example reconstruction scale changes, or because token-sum changes which examples dominate gradients?
- Does
zfree_reference_scalefully cancel the aggregation-scale change for normalized objectives, or only make final scalar magnitudes comparable? - Does high final answer accuracy hide a posterior-router mismatch where the generator solves tasks while router samples carry weak strategy information?
- Do sum runs lose router structure early, or do they initially learn structure and collapse later?
- Does tokcc under sum focus on a narrow token subset, leaving raw sequence reconstruction poor but final answers intact?
- Are mean-reference comparisons confounded by schedule differences because sum uses constant beta while matched mean rows use
lin100?
Local Interpretation
The inferred zfree_reference_scale is 40.1x larger for sum than for the matched mean-reference rows. That is expected if z-free reference-scale estimation follows the same token aggregation mode as training, but it means raw reconstruction loss values are not directly comparable across aggregation modes. The active objective and effective KL-pressure figures should therefore be preferred over raw loss comparisons when forming hypotheses from local data.
Dynamics Interpretation
The validation dynamics make the earlier final-state diagnosis sharper. At the first available validation checkpoint, step 31, the median base-relative reconstruction across sum rows is 20.7399; by the final checkpoint, step 1563, the median is 1.0008. This indicates that most non-tokcc sum runs rapidly move from very poor base-relative reconstruction toward the z-free baseline, but do not continue to a clearly better-than-baseline regime comparable to the strongest sequence-mean reference rows.
The KL trajectories also look like collapse or severe compression under many settings: median validation KL drops from 147.5033 at the first checkpoint to 1.2447 at the final checkpoint, while median final validation matched-router score remains 0.2936. This supports the hypothesis that the generator can recover high answer accuracy while the learned latent/router pathway carries limited strategy information.
The caveat is that these are validation checkpoints, not exact initialization values, and the first checkpoint is already after training has begun. The figures therefore answer the training-dynamics question but not the exact initialization-scale question. They also still cannot tell whether token-sum changes which examples or token positions dominate gradients; that requires per-example or token-count contribution aggregates.
Base-Relative Reconstruction Interpretation
The base-relative reconstruction versus within-task router-probe figure suggests that the sum rows mostly fail to improve raw/base-relative reconstruction beyond the z-free reference model. For the sum rows that directly optimize raw or distribution-normalized reconstruction, distribution_normalized_reconstruction_loss stays close to 1.0 rather than moving substantially below it: raw sum is about 0.9975-0.9979, dist-norm sum is about 0.9966-1.0001, and dist+tokcc sum is about 0.9978-1.0072. This is qualitatively different from the strongest matched mean-reference cell, dist_norm beta=.01, where base-relative reconstruction is 0.3761 and within-task router probe is 0.7600.
One interpretation is that token-sum aggregation preserves base-level likelihood without forcing useful strategy-conditioned reconstruction. The z-free baseline is already strong, final answer accuracy is saturated, and the generator may be able to solve the tasks mostly from x without using router-sampled z in a strategy-informative way. This would explain why base-relative reconstruction hovers near 1.0 while within-task router probe remains near chance-to-weak levels.
A second possibility is that token-sum changes the optimization geometry by giving more mass to long or easy supervised-token regions. If the summed loss is dominated by tokens that are predictable without strategy information, the model can reduce or maintain aggregate likelihood while learning little about the latent strategy variable. The current aggregate cannot distinguish this from ordinary reconstruction saturation; resolving it requires per-task or per-example token-count and loss-contribution aggregates.
The tokcc rows should be interpreted separately. Their base-relative reconstruction is very high, around 18-19, but tokcc rows do not optimize base-relative/raw reconstruction as their active objective; they optimize the concentration-corrected token-weighted reconstruction term, which stays near 1.026. That makes the high base-relative value evidence that tokcc is not preserving raw/base-relative likelihood under sum, but not direct evidence that the active tokcc objective itself failed to optimize.
The main caution is confounding: the sum rows use constant beta, while the matched mean-reference rows use lin100 schedules and lack beta 1.0 common cells. The figure is therefore best read as a failure diagnosis for this local sum grid, not as a clean causal estimate of token aggregation mode. The new validation dynamics show that most non-tokcc rows move toward the z-free baseline after the first available checkpoint; the remaining unresolved questions are exact step-0 scale and whether per-example or token-count loss contributions explain why the sum objective does not produce stronger router structure.
