Sum Token Aggregation Ablation: Results
Sum Token Aggregation Ablation: Results
Note
This file is observational. Interpretive claims and recommendations belong in analysis.
Collection State
The collection has 12 expected cells: 12 have final metrics and 0 remain without final metrics.
| status | cells |
|---|---|
| complete | 12 |
| incomplete | 0 |
Aggregate Metrics
Rows vary reconstruction family and beta while fixing token aggregation to sum.
The best sum row by matched router score is tokcc a.25 at beta=.01 with matched router score 0.5502, within-task router probe 0.5521, and final answer accuracy 0.9971.
Metric Distributions
| metric | min | mean | median | max | std |
|---|---|---|---|---|---|
| matched router score | 0.4929 | 0.5216 | 0.5211 | 0.5502 | 0.0148 |
| within-task router probe | 0.4942 | 0.5229 | 0.5225 | 0.5521 | 0.015 |
| global router probe | 0.1191 | 0.147 | 0.1483 | 0.1713 | 0.0156 |
| strategy-match rate | 0.996 | 0.9975 | 0.9976 | 0.9983 | 0.0006 |
| posterior probe | 0.6854 | 0.7647 | 0.7808 | 0.799 | 0.0344 |
| final answer accuracy | 0.9964 | 0.9977 | 0.9978 | 0.9984 | 0.0005 |
| reconstruction loss | 1.1447 | 6.2324 | 1.1476 | 21.8072 | 8.8077 |
| dist-normalized reconstruction | 0.9966 | 5.4261 | 0.9991 | 18.9858 | 7.6682 |
| tokcc reconstruction | 1.0246 | 1.026 | 1.0256 | 1.0296 | 0.0014 |
| KL to prior | 0.0057 | 1.566 | 1.563 | 3.104 | 1.0799 |
| active reconstruction objective | 0.9966 | 1.0486 | 1.0252 | 1.1462 | 0.0572 |
| KL / active recon | 0.005 | 1.5032 | 1.5246 | 3.0332 | 1.0486 |
| inferred z-free reference scale | 1.1486 | 1.1486 | 1.1486 | 1.1486 | 0 |
| posterior-router probe gap | 0.1763 | 0.2417 | 0.2464 | 0.2873 | 0.0322 |
Mean-Versus-Sum Common Cells
The table below compares the eight sum rows with matching Qwen2.5 0.5B z64 sequence-mean rows from the May 1 paper-method grid. The beta 1.0 sum rows are omitted from this contrast because there is no matching row in that reference table.
| reconstruction_loss_family_label | beta_label | mean_matched_router_score | sum_matched_router_score | delta_matched_router_score | mean_final_answer_accuracy | sum_final_answer_accuracy | delta_final_answer_accuracy | mean_active_reconstruction_objective | sum_active_reconstruction_objective | delta_active_reconstruction_objective |
|---|---|---|---|---|---|---|---|---|---|---|
| raw | beta=.01 | 0.5719 | 0.5283 | -0.0437 | 0.9995 | 0.9978 | -0.0017 | 0.032 | 1.1462 | 1.1141 |
| raw | beta=.1 | 0.565 | 0.5165 | -0.0486 | 0.9995 | 0.998 | -0.0015 | 0.032 | 1.1457 | 1.1137 |
| dist norm | beta=.01 | 0.7575 | 0.5427 | -0.2149 | 0.998 | 0.9979 | -0.0001 | 0.3761 | 0.9966 | 0.6205 |
| dist norm | beta=.1 | 0.573 | 0.511 | -0.062 | 0.9995 | 0.9984 | -0.0011 | 1.1166 | 0.9988 | -0.1178 |
| tokcc a.25 | beta=.01 | 0.7952 | 0.5502 | -0.245 | 0.9938 | 0.9971 | 0.0033 | 0.1115 | 1.0256 | 0.9141 |
| tokcc a.25 | beta=.1 | 0.592 | 0.5115 | -0.0805 | 0.9996 | 0.9975 | -0.0021 | 1.1532 | 1.0256 | -0.1276 |
| dist+tokcc 0.05/0.95 | beta=.01 | 0.577 | 0.521 | -0.0561 | 0.9991 | 0.9976 | -0.0015 | 1.1518 | 1.0235 | -0.1282 |
| dist+tokcc 0.05/0.95 | beta=.1 | 0.5686 | 0.4929 | -0.0757 | 0.9994 | 0.9975 | -0.0019 | 1.1518 | 1.0233 | -0.1285 |
Mean-Versus-Sum Objective Scale
The heatmap annotates true sum-minus-mean deltas. Colors are row-normalized so reconstruction-scale deltas and router-score deltas can share one panel.
Local Diagnostics
KL-To-Reconstruction Trade-Off
KL/Reconstruction Ratio
Base-Relative Reconstruction Versus Router Probe
Base-relative reconstruction is distribution_normalized_reconstruction_loss, i.e. raw reconstruction divided by that run's inferred z-free reference scale.
Inferred Z-Free Reference Scale
Inferred as reconstruction_loss / distribution_normalized_reconstruction_loss. The sum scale is 40.1x the matched mean-reference scale.
| aggregation_mode | source | n | min | mean | max | std |
|---|---|---|---|---|---|---|
| sum | sum summary.csv | 12 | 1.1486 | 1.1486 | 1.1486 | 0 |
| mean_reference_common_cells | May 1 final_05b_summary.csv | 8 | 0.0287 | 0.0287 | 0.0287 | 0 |
| sum_over_mean_reference_ratio | derived from rows above | 1 | 40.0771 | 40.0771 | 40.0771 | 0 |
Posterior-Router Gap
Validation Dynamics
These dynamics are validation metrics from data/sum_eval_history_long.csv, not final test metrics. The earliest available checkpoint is step 31, so this is a first-eval view rather than an exact initialization view.
Base-Relative Reconstruction Dynamics
Active Reconstruction Objective Dynamics
KL Dynamics
KL/Reconstruction Ratio Dynamics
Router And Posterior Probe Dynamics
Reconstruction-Router Trajectory
Per-Task Router Dynamics
Dynamics Summary Table
| reconstruction_loss_family_label | beta_label | first_global_step | final_global_step | first_base_relative_reconstruction | min_base_relative_reconstruction | final_base_relative_reconstruction | first_kl_prior_loss | final_kl_prior_loss | max_matched_router_score | final_matched_router_score |
|---|---|---|---|---|---|---|---|---|---|---|
| raw | beta=.01 | 31 | 1563 | 16.4118 | 0.9977 | 0.9978 | 214.284 | 1.8825 | 0.449 | 0.294 |
| raw | beta=.1 | 31 | 1563 | 19.5004 | 0.999 | 0.999 | 155.467 | 0.0048 | 0.4199 | 0.2679 |
| raw | beta=1 | 31 | 1563 | 21.478 | 0.9998 | 1.0001 | 135.055 | 1.1542 | 0.3531 | 0.3041 |
| dist norm | beta=.01 | 31 | 1563 | 16.6022 | 0.9989 | 0.9999 | 204.706 | 1.925 | 0.4579 | 0.2806 |
| dist norm | beta=.1 | 31 | 1563 | 19.8601 | 0.9997 | 1.0009 | 153.902 | 0.0046 | 0.4078 | 0.2651 |
| dist norm | beta=1 | 31 | 1563 | 21.445 | 1.001 | 1.0025 | 142.268 | 0.0113 | 0.3609 | 0.2874 |
| tokcc a.25 | beta=.01 | 31 | 1563 | 20.2857 | 18.1475 | 18.6392 | 157.158 | 1.3352 | 0.4333 | 0.3095 |
| tokcc a.25 | beta=.1 | 31 | 1563 | 21.89 | 18.2274 | 19.0668 | 149.055 | 1.4631 | 0.4131 | 0.2886 |
| tokcc a.25 | beta=1 | 31 | 1563 | 22.8342 | 18.1651 | 18.7064 | 145.952 | 1.0142 | 0.4178 | 0.2949 |
| dist+tokcc 0.05/0.95 | beta=.01 | 31 | 1563 | 18.7655 | 0.9995 | 1 | 133.978 | 2.0311 | 0.4311 | 0.2936 |
| dist+tokcc 0.05/0.95 | beta=.1 | 31 | 1563 | 21.194 | 1.0001 | 1.0007 | 137.821 | 2.9213 | 0.4046 | 0.2987 |
| dist+tokcc 0.05/0.95 | beta=1 | 31 | 1563 | 22.4017 | 1.0108 | 1.0108 | 139.672 | 1.0344 | 0.3972 | 0.2937 |
Top Per-Task Validation Router Checkpoints
| task_name | reconstruction_loss_family_label | beta_label | final_matched_router_score | max_matched_router_score | final_final_answer_accuracy |
|---|---|---|---|---|---|
| grid_pathfinding | dist norm | beta=1 | 0.8991 | 0.9619 | 1 |
| grid_pathfinding | tokcc a.25 | beta=.01 | 0.8761 | 0.9554 | 1 |
| grid_pathfinding | dist norm | beta=.1 | 0.8362 | 0.9541 | 1 |
| grid_pathfinding | dist+tokcc 0.05/0.95 | beta=.1 | 0.8879 | 0.9537 | 1 |
| grid_pathfinding | raw | beta=1 | 0.8898 | 0.9469 | 1 |
| grid_pathfinding | tokcc a.25 | beta=1 | 0.8718 | 0.9423 | 1 |
| grid_pathfinding | dist+tokcc 0.05/0.95 | beta=.01 | 0.8525 | 0.9364 | 1 |
| grid_pathfinding | raw | beta=.01 | 0.8655 | 0.9352 | 1 |
| grid_pathfinding | tokcc a.25 | beta=.1 | 0.8678 | 0.9352 | 1 |
| grid_pathfinding | dist norm | beta=.01 | 0.8699 | 0.934 | 1 |
| grid_pathfinding | dist+tokcc 0.05/0.95 | beta=1 | 0.9167 | 0.9229 | 1 |
| grid_pathfinding | raw | beta=.1 | 0.8803 | 0.9194 | 1 |
Derived Local Diagnostic Table
| reconstruction_loss_family_label | beta_label | matched_router_score | active_reconstruction_objective | kl_prior_loss | kl_over_active_reconstruction | inferred_zfree_reference_scale | posterior_router_probe_gap |
|---|---|---|---|---|---|---|---|
| raw | beta=.01 | 0.5283 | 1.1462 | 2.4084 | 2.1013 | 1.1486 | 0.2181 |
| raw | beta=.1 | 0.5165 | 1.1457 | 0.0057 | 0.005 | 1.1486 | 0.2083 |
| raw | beta=1 | 0.5213 | 1.1459 | 1.1119 | 0.9703 | 1.1486 | 0.2566 |
| dist norm | beta=.01 | 0.5427 | 0.9966 | 2.2814 | 2.2892 | 1.1486 | 0.2363 |
| dist norm | beta=.1 | 0.511 | 0.9988 | 0.0072 | 0.0072 | 1.1486 | 0.211 |
| dist norm | beta=1 | 0.5081 | 1.0001 | 0.0101 | 0.0101 | 1.1486 | 0.1763 |
| tokcc a.25 | beta=.01 | 0.5502 | 1.0256 | 1.4973 | 1.4599 | 1.1486 | 0.2301 |
| tokcc a.25 | beta=.1 | 0.5115 | 1.0256 | 2.6892 | 2.622 | 1.1486 | 0.275 |
| tokcc a.25 | beta=1 | 0.528 | 1.0268 | 1.2416 | 1.2092 | 1.1486 | 0.2688 |
| dist+tokcc 0.05/0.95 | beta=.01 | 0.521 | 1.0235 | 2.8067 | 2.7422 | 1.1486 | 0.264 |
| dist+tokcc 0.05/0.95 | beta=.1 | 0.4929 | 1.0233 | 3.104 | 3.0332 | 1.1486 | 0.2873 |
| dist+tokcc 0.05/0.95 | beta=1 | 0.5276 | 1.0248 | 1.6287 | 1.5893 | 1.1486 | 0.2691 |
Top And Bottom Rows
Top Rows By Matched Router Score
| reconstruction_loss_family_label | beta_label | matched_router_score | within_task_router_probe | strategy_match_rate | final_answer_accuracy | reconstruction_loss | kl_prior_loss |
|---|---|---|---|---|---|---|---|
| tokcc a.25 | beta=.01 | 0.5502 | 0.5521 | 0.9966 | 0.9971 | 21.2866 | 1.4973 |
| dist norm | beta=.01 | 0.5427 | 0.5439 | 0.9978 | 0.9979 | 1.1447 | 2.2814 |
| raw | beta=.01 | 0.5283 | 0.5294 | 0.9978 | 0.9978 | 1.1462 | 2.4084 |
| tokcc a.25 | beta=1 | 0.528 | 0.5302 | 0.996 | 0.9964 | 21.3657 | 1.2416 |
| dist+tokcc 0.05/0.95 | beta=1 | 0.5276 | 0.5289 | 0.9976 | 0.9978 | 1.1569 | 1.6287 |
Bottom Rows By Matched Router Score
| reconstruction_loss_family_label | beta_label | matched_router_score | within_task_router_probe | strategy_match_rate | final_answer_accuracy | reconstruction_loss | kl_prior_loss |
|---|---|---|---|---|---|---|---|
| dist+tokcc 0.05/0.95 | beta=.1 | 0.4929 | 0.4942 | 0.9974 | 0.9975 | 1.148 | 3.104 |
| dist norm | beta=1 | 0.5081 | 0.5091 | 0.9981 | 0.9982 | 1.1488 | 0.0101 |
| dist norm | beta=.1 | 0.511 | 0.5119 | 0.9983 | 0.9984 | 1.1472 | 0.0072 |
| tokcc a.25 | beta=.1 | 0.5115 | 0.513 | 0.997 | 0.9975 | 21.8072 | 2.6892 |
| raw | beta=.1 | 0.5165 | 0.5175 | 0.998 | 0.998 | 1.1457 | 0.0057 |
Repo-Side Artifacts
- Aggregate directory:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503 - Report:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/report.md - Brief:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/brief.yaml - Findings JSON:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/findings.json - Notebook:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/notebook.ipynb - Local diagnostics table:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/local_diagnostics_summary.csv - Data availability table:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/diagnostic_data_availability.csv - Base-relative reconstruction/router data:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/base_relative_reconstruction_vs_router.csv - Sum validation dynamics:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/sum_eval_history_long.csv - Sum per-task validation dynamics:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/sum_eval_history_per_task_long.csv - Sum dynamics summary:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/sum_dynamics_summary.csv - Sum per-task dynamics summary:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/synthetic-pretrained-disentangled-from-entangled-v2-reconstruction-token-aggregation-sum-ablation-multi-task-z64-qwen25-05b_20260503/data/sum_per_task_dynamics_summary.csv
