TOK-CC Deep Dive: Reconstruction Frontier Anomaly
TOK-CC Deep Dive: Reconstruction Frontier Anomaly
This note examines why tokcc_a0p25 has high matched-strategy router score while its baseline-normalized raw reconstruction loss is much larger than the other high-performing reconstruction families in the Apr27 paper basic-grid snapshot.
Setup
The source aggregate is path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429. The analysis uses the 180 complete single-task runs and excludes the separate complete multi_task slice, which is analyzed in multitask.
The key implementation detail is that analysis/final_baseline_normalized_reconstruction_loss is a raw reconstruction ratio, not a bounded [0, 1] information measure. In the current metrics it is computed as raw reconstruction divided by a z-free reference scale; values larger than 1 are valid. Across the complete Apr27 rows, the test-scale proxy loss/test/reconstruction_loss / loss/test/distribution_normalized_reconstruction_loss has mean 0.0318 and standard deviation 0.0132.
At the family level, tokcc_a0p25 has mean baseline-normalized raw reconstruction 11.3377 and mean matched-strategy router score 0.7308, while its own token-concentration-corrected reconstruction component is 0.3213. The hybrid dist_plus_tokcc_a0p25 has mean baseline-normalized raw reconstruction 0.3622 and matched score 0.7163.
Family Summary
| family | n | base_norm_raw | token_cc | active_obj | router_probe | strategy_match | matched_score | final_acc |
|---|---|---|---|---|---|---|---|---|
| raw | 36 | 0.9226 | 0.9349 | 0.0278 | 0.504 | 0.9961 | 0.5021 | 0.9975 |
| dist_norm | 36 | 0.3695 | 0.3722 | 0.3695 | 0.6971 | 0.9958 | 0.6939 | 0.9973 |
| example_inv_surp | 36 | 0.3802 | 0.3837 | 0.3437 | 0.6917 | 0.9962 | 0.6889 | 0.9978 |
| tokcc_a0p25 | 36 | 11.3377 | 0.3213 | 0.3213 | 0.7401 | 0.9886 | 0.7308 | 0.995 |
| dist_plus_tokcc_a0p25 | 36 | 0.3622 | 0.3652 | 0.3635 | 0.7206 | 0.9947 | 0.7163 | 0.9969 |
The heatmap uses column-standardized color and raw mean annotations, so each column should be read as a within-metric comparison across families. It shows the same split numerically: tokcc_a0p25 is extreme on raw reconstruction views, not on the token-corrected reconstruction objective or the router metrics.
Component Frontier Views
The baseline-normalized raw reconstruction panel makes tokcc_a0p25 look anomalous because that panel plots raw reconstruction scale. The token-level panel shows the complementary view: tokcc_a0p25 is no longer an obvious reconstruction outlier when judged by the component it actually optimizes.
The active-objective view uses the weighted reconstruction component from each loss setting. This is still not a universal loss scale, but it is the correct diagnostic for asking whether tokcc_a0p25 optimized the objective it was given.
This comparison separates the raw reconstruction ratio from the normalized concentration-corrected token objective. tokcc_a0p25 is high on the raw ratio and low on the TOK-CC component, which is exactly the metric/objective mismatch predicted by the loss definition.
Hybrid Control
Matched task-level comparisons show what happens when the distribution-normalized raw reconstruction term is added back. Averaged over matched task, beta, and schedule cells, tokcc_a0p25 - dist_plus_tokcc_a0p25 changes matched score by 0.0145 but changes baseline-normalized raw reconstruction by 10.9755. This supports using the hybrid as a control: it preserves much of the TOK-CC router benefit while removing the raw reconstruction-scale anomaly.
Probe Decomposition
The matched score is the product of router probe accuracy and strategy-match rate. In this grid, final answer accuracy and strategy-match rate are high across many settings, so most of the family separation comes from router probe accuracy rather than from marginal task success.
Top Loss Settings
| loss_setting | n | base_norm_raw | token_cc | active_obj | matched_score |
|---|---|---|---|---|---|
| tokcc_a0p25_b0p01_lin100 | 6 | 7.8127 | 0.0313 | 0.0313 | 0.8811 |
| example_inv_surp_b0p01_lin100 | 6 | 0.0487 | 0.0417 | 0.0396 | 0.8789 |
| tokcc_a0p25_b0p1_lin100 | 6 | 10.883 | 0.049 | 0.049 | 0.8651 |
| dist_plus_tokcc_a0p25_b0p01_lin100 | 6 | 0.0634 | 0.0501 | 0.0558 | 0.8648 |
| dist_norm_b0p01_lin100 | 6 | 0.0172 | 0.0114 | 0.0173 | 0.8605 |
| dist_plus_tokcc_a0p25_b0p1_lin100 | 6 | 0.0452 | 0.0355 | 0.039 | 0.85 |
| dist_plus_tokcc_a0p25_b0p01_const | 6 | 0.0185 | 0.0142 | 0.0182 | 0.8479 |
| tokcc_a0p25_b0p01_const | 6 | 9.3189 | 0.0364 | 0.0364 | 0.8471 |
Correlation Checks
| group | component | n | pearson_corr_with_matched_score |
|---|---|---|---|
| all | baseline_normalized_raw_reconstruction | 180 | 0.0373 |
| all | token_cc_reconstruction | 180 | -0.9201 |
| all | active_reconstruction_objective_loss | 180 | -0.6011 |
| tokcc_a0p25 | baseline_normalized_raw_reconstruction | 36 | -0.2552 |
| tokcc_a0p25 | token_cc_reconstruction | 36 | -0.8789 |
| tokcc_a0p25 | active_reconstruction_objective_loss | 36 | -0.8789 |
Working Hypotheses
Metric/objective mismatch is the strongest explanation. tokcc_a0p25 optimizes the normalized concentration-corrected token-weighted reconstruction term, not the raw baseline-normalized reconstruction ratio. Its high raw ratio says that broad teacher-forced raw sequence likelihood remains poor relative to the z-free scale; it does not by itself say that the latent variable lacks strategy information.
Strategy information appears more localized than full-output reconstruction. A router probe can succeed if z carries the strategy-discriminative aspects of y, even when the full raw reconstruction loss is high. The current aggregate does not expose token-level per-position attribution, so this remains an indirect inference from the gap between the raw ratio and the token-corrected objective.
The hybrid control is paper-useful. dist_plus_tokcc_a0p25 keeps matched score near the TOK-CC-only family while reducing baseline-normalized raw reconstruction by an order of magnitude. This makes it a cleaner method candidate when the paper wants both router factorization and a reconstruction-frontier story.
The [0, 1] intuition should be revised. Baseline-normalized reconstruction is better described as a ratio to the latent-free reference model, not a clipped information score. Values above 1 are meaningful and occur systematically for tokcc_a0p25 here.
Implications For Paper Figures
The reconstruction-frontier claim should avoid treating baseline-normalized raw reconstruction as the single universal x-axis across all reconstruction objectives. A safer figure package is: use distribution/example/token component frontiers to show objective-specific behavior; use the active-objective frontier to explain tokcc_a0p25; and use dist_plus_tokcc_a0p25 when the text needs a single method that combines low raw reconstruction ratio with strong router factorization.
The main paper claim can still say that reconstruction shaping improves decodable routing, but the TOK-CC result suggests a sharper statement: a loss can improve strategy-decodable routing by emphasizing the strategy-relevant parts of reconstruction even when raw sequence-level reconstruction remains worse.
Repo-Side Artifacts
- Deep-dive artifact directory:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive - Deep-dive findings JSON:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive/findings.json - Deep-dive manifest:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive/aggregate_manifest.json - Run-level diagnostics:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive/data/tokcc_deepdive_run_metrics.csv - Family summary:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive/data/tokcc_deepdive_family_summary.csv - Paired TOK-CC versus hybrid deltas:
path:experiments/synthetic_sequences/disentangled/analysis/aggregates/exp2_continuous_cvae-Apr27-paper-basic-grid_20260429/tokcc_deepdive/data/tokcc_hybrid_paired_deltas.csv
