continuous_cvae-Apr24-reconstruction-stack_20260424 Design

Back to hub

Summary

continuous_cvae-Apr24-reconstruction-stack_20260424 is a continuous-CVAE follow-up to continuous_cvae-Apr22-beta-schedule-followup_20260422. It preserves the Apr22 task grid, adaptation modes, latent dimension, optimizer, one-epoch training budget, and single seed, but changes the objective menu from schedule-only tuning to a five-term reconstruction-loss configuration.

Planned design:

• 6 tasks x 13 loss settings x 3 adaptation modes x 1 seed = 234 jobs
• latent type continuous, continuous_latent_dim=8, last-token pooling
• adaptation modes full, lora, reinit
• optimizer muon, lr=1e-3, 1 epoch, max_sequences_per_epoch=200000
• supervision source groundtruth-data
• reconstruction-stack settings from path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_sweep.py

Relevant repo-side definitions:

• Slurmkit collection: path:.slurmkit/collections/synthetic_sequences/families/disentangled_from_entangled_random_init/continuous_cvae_apr24_reconstruction_stack.yaml
• Family spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/specs/continuous_cvae_apr24_reconstruction_stack.yaml
• Parser: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/parse_params_continuous_apr24_reconstruction_stack.py
• Shared reconstruction menu: path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_sweep.py

Current Collection State

As of the latest analysis rebuild on 2026-04-26, the repo-side aggregate has 234/234 complete unique artifacts, with 0 running cells, 0 pending cells, 0 completed cells missing artifacts, and 0 duplicate-artifact cells excluded from scientific figures. The local run directory still contains more run directories than the planned grid because it includes earlier or non-grid artifacts; the analysis treats the slurmkit job grid as the authoritative expected cell list, then left-joins discovered run artifacts onto that grid.

The current run identity appears to include the new reconstruction fields: sampled manifests contain raw_reconstruction_weight, distribution_normalized_reconstruction_weight, example_inverse_surprisal_reconstruction_weight, normalized_concentration_corrected_token_weighted_reconstruction_weight, and token_weighted_reconstruction_alpha, and sanitized_run_config records run_identity_schema_version=2. However, run_manifest.json does not consistently include the original loss_setting string, so the analysis must derive loss_setting, loss_setting_family, and beta_schedule_setting from the explicit reconstruction and beta fields or from the slurmkit job parameters.

Why This Collection Exists

Apr22 showed that beta timing alone matters: delayed-linear schedules, especially beta0p1_d50_lin50, gave the strongest average router-probe-by-accuracy tradeoff, while beta0p01_cyclin25_w12p5 remained a competitive second candidate. Apr22 also left the reconstruction objective mostly fixed. This Apr24 collection asks whether the reconstruction term itself can improve the same frontier.

The collection is also positioned against the paper claims in claims:

• C3 needs evidence that base-normalized reconstruction and strategy decodability move together, and that raw ELBO-like settings sit in a collapsed or dominated region.
• C4 needs matched evidence about how full, lora, and reinit respond differently under the same objective.
• C5 needs an empirical anchor for the standard-ELBO failure mode: preserved marginal fit is not sufficient for a strategy-bearing latent.

Sweep Coverage

Tasks:

• base_conversion
• grid_pathfinding
• linear_equation_solving
• list_summation
• multidigit_addition
• sorting_algorithms

Adaptation modes:

• full
• lora
• reinit

Loss settings:

The raw rows are beta-1 ELBO-style baselines. The method rows hold beta at 0.01 and compare reconstruction variants under constant and delayed-linear timing.

Research Questions

1. Do distribution-level H(Y|X)-normalized reconstruction losses improve the router-decoding and accuracy frontier relative to raw beta-1 ELBO-style baselines?
2. Does example-level surprisal normalization add useful pressure beyond distribution-level normalization, or does it trade away accuracy, strategy coverage, or router-side decodability?
3. Does normalized concentration-corrected token weighting improve the frontier beyond distribution-level normalization alone, and does token alpha 0.5 behave more stably than alpha 1.0?
4. Does the Apr22 delayed-linear advantage persist when the reconstruction family changes, or are some reconstruction variants better with constant low beta?
5. Are reconstruction-family effects consistent across full, lora, and reinit, or do adapted-from-entangled and random-reinit regimes require different objectives?

Falsifiable Hypotheses

H1. Raw beta-1 baselines will preserve final-answer accuracy better than they recover router-side strategy information, placing them near the ELBO-failure region described in claim C5.

H2. Distribution-normalized reconstruction at beta=0.01 will improve the composite router probe x final accuracy score relative to raw beta-1 baselines, because it directly optimizes a base-relative reconstruction scale.

H3. Example inverse-surprisal normalization will change the task balance: it may help tasks where a few high-surprisal examples dominate raw loss, but it may hurt if it underweights hard examples that carry strategy-discriminative information.

H4. Adding normalized concentration-corrected token weighting will improve router-side probe accuracy when strategy information is localized in a small subset of output tokens; alpha 0.5 should be less brittle than alpha 1.0 if full concentration overweights narrow token positions.

H5. Delayed-linear beta timing will remain better than constant beta for the composite score in most method rows, matching the Apr22 schedule-only result, but this should be tested within each reconstruction family rather than assumed globally.

H6. full and lora will preserve accuracy more reliably than reinit, while reinit may still produce high latent separability on tasks where optimization reaches a good basin.

Primary Metrics

• sampled_test_final/router_sampled_z/linear_probe_accuracy: primary router-side latent strategy decodability.
• sampled_test_final/final_answer_accuracy: primary task-performance guardrail.
• analysis/router_probe_times_strategy_match_rate: derived composite used for headline ranking; this follows the Apr22 dynamics metric labeled "matched-strategy router score."
• continuous_diagnostics/test_final/posterior_mu/linear_probe_accuracy: posterior-side decodability, used to separate inference-side structure from router-side usability.
• loss/val/baseline_normalized_reconstruction_loss or final-eval equivalent when available: base-relative reconstruction ratio for Figure 4-style analysis.

Secondary metrics:

• sampled_test_final/strategy_match_rate
• sampled_test_final/unique_strategy_rate
• sampled_test_final/parse_success_rate
• sampled_test_final/invalid_structure_rate
• sampled_test_final/router_variance_mean
• loss/test/reconstruction_loss
• loss/test/distribution_normalized_reconstruction_loss
• loss/test/example_inverse_surprisal_weighted_reconstruction_loss
• loss/test/normalized_concentration_corrected_token_weighted_reconstruction_loss
• loss/test/kl_prior_loss
• profile_total/total_seconds

Figure Plan

F0. Run-identity and completeness audit. Heatmap of expected task x loss_setting x adaptation_mode cells, with states for missing, running, complete, and collision-suspect. This is a gate figure, not a scientific result.

F1. Global loss-setting summary. Grouped bars or dot plots for final accuracy, router probe, strategy match rate, posterior probe, baseline-normalized reconstruction, and the matched-strategy router score by loss setting, split by adaptation mode.

F2. Reconstruction-family x beta-schedule heatmaps. Heatmaps over reconstruction family and beta schedule for each headline metric, faceted by adaptation mode and optionally task.

F3. Matched contrast ladder. Paired deltas for: delayed-linear minus constant within each method family; example inverse-surprisal minus distribution-normalized; token-weighted stack minus distribution-normalized; alpha 1.0 minus alpha 0.5; and raw delayed schedules versus raw constant.

F4. Reconstruction ratio versus decodability scatter. One point per valid run, x-axis base-normalized reconstruction ratio, y-axis router probe, color by reconstruction family, shape by adaptation mode. This is the direct evidence candidate for Figure 4.

F5. Adaptation-mode paired comparison. Matched-setting scatter or delta plot comparing reinit against full and lora on router probe, accuracy, and reconstruction ratio. This is the direct evidence candidate for Figure 5.

F6. Training dynamics for selected settings. Validation curves for final accuracy, router probe, posterior probe, reconstruction ratio, component losses, and KL for a small set of representative settings: best composite, best router-only, best raw baseline, and any collapsed setting.

F7. Objective component scale audit. Loss component values and weighted contributions by reconstruction family, to show whether a variant wins because it improves the intended component or because component scales changed unexpectedly.

F8. Top and bottom run cards with latent snapshots. Compact tables plus router-sampled and posterior-mu latent-space frames for top and failure cases.

Analysis Plan And Missing-Run Handling

The repo-side analysis should be idempotent and partial-grid aware:

1. Build expected_cells.csv from the slurmkit collection jobs, with one row per planned (task_name, loss_setting, adaptation_mode, seed) cell plus all resolved loss weights, beta schedule fields, job ids, and latest slurm state.
2. Discover run artifacts under path:.runs/synthetic_sequences/disentangled/exp2_continuous_cvae-Apr24-reconstruction-stack, but do not use directory count as the denominator. A run is analyzable only when run_manifest.json and metrics.json exist and its manifest fields match exactly one expected cell.
3. Derive missing loss_setting values from exact reconstruction-weight and beta-schedule signatures, using LOSS_SETTINGS from path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_sweep.py. The aggregate code should preserve the explicit component weights as separate columns.
4. Left-join artifacts onto expected cells and assign a status column: complete, running, pending, failed, missing_artifact, duplicate_artifact, or unmatched_artifact. Scientific figures should filter to complete; completeness figures should show all statuses.
5. Generate aggregate summaries over available completed runs, always reporting n_complete and n_expected for each group. Any mean, delta, or ranking must carry its denominator so partial snapshots are not mistaken for complete-grid evidence.
6. Save outputs under path:experiments/synthetic_sequences/disentangled/analysis/aggregates/continuous_cvae-Apr24-reconstruction-stack_20260424, including summary.csv, metrics_long.csv or beta_sweep_long.csv, eval_history_long.csv, expected_cells.csv, missing_or_incomplete_cells.csv, unmatched_artifacts.csv, aggregate_manifest.json, findings.json, and report.md.
7. Re-running the analysis after additional jobs finish should require only a force rebuild of aggregates and figure regeneration; no hand-maintained missing-run list should be needed.

Empirical Claims Requiring Evidence

1. Raw beta-1 ELBO-style rows are dominated by at least one normalized reconstruction variant on router-side decodability at matched or acceptable final-answer accuracy. Evidence: F1, F3, F4, top-run tables.
2. Distribution-level normalization improves the reconstruction-decodability frontier relative to raw reconstruction. Evidence: F1, F2, F4, distribution-normalized versus raw contrasts.
3. Example-level surprisal normalization either adds value beyond distribution normalization or produces a measurable tradeoff that argues against using it alone. Evidence: F2, F3, task-level deltas, accuracy guardrail tables.
4. Normalized concentration-corrected token weighting improves, worsens, or leaves unchanged the frontier relative to distribution normalization alone; the alpha comparison must be reported separately. Evidence: F2, F3, alpha 1.0 versus alpha 0.5 deltas.
5. Delayed-linear beta timing has a consistent positive, negative, or family-dependent effect under the new reconstruction stack. Evidence: F2, F3, training dynamics in F6.
6. Posterior-mu decodability remains near ceiling while router-sampled decodability varies, or the collection shows a genuine posterior-side degradation for specific variants. Evidence: F1, F2, F6.
7. full, lora, and reinit have different objective sensitivities at matched settings. Evidence: F1, F5, per-mode top and bottom tables.
8. Paper-facing claims about ELBO failure or reconstruction-decoding correlation are restricted to complete, uniquely matched artifacts and report denominators for partial snapshots. Evidence: F0, aggregate manifest, unmatched-artifact table.

Success Criteria

• Every planned cell has a unique, auditable mapping from intended loss_setting to run artifact.
• The aggregate summary can recover all new reconstruction-weight fields from a trusted source, ideally run_manifest.json; if not, from a one-to-one slurm job script/log mapping with no output collisions.
• The headline ranking uses analysis/router_probe_times_strategy_match_rate, with final-answer accuracy still shown as a non-negotiable guardrail.
• The report can state which reconstruction variant, if any, should replace Apr22's schedule-only candidate for future runs.
• The analysis produces at least one credible Figure 4 candidate and one credible Figure 5 candidate for the paper storyboard.

Known Risks And Confounders

• The collection uses one seed, so task- and mode-level patterns are directional rather than variance-estimated.
• Raw baselines use beta=1, while method rows use beta=0.01; raw-vs-method comparisons therefore mix reconstruction-family and beta-strength differences. The cleaner within-method contrast is constant versus delayed-linear at fixed beta=0.01. This design choice was made because beta=1 reflects a "standard method" (standard ELBO), which is useful baseline for the purposes of a research paper.
• Component-loss scales changed in the refactor. The analysis should show both unweighted component values and weighted contributions before interpreting a variant as "better."
• The collection now has a complete repo-side aggregate, but it is still a single-seed sweep; global means and rankings should be treated as directional until replicated.
• loss_setting is not always persisted directly in run manifests, so the analysis must verify the derived setting against the explicit reconstruction weights and loss_tag.
• Runtime is secondary; profiling can explain analysis cost and objective overhead but should not drive the scientific ranking unless quality metrics are tied.

Immediate Guidance Questions

1. Should partial-snapshot scientific figures be generated now with explicit denominators, or should the first report include only the completeness and data-integrity audit until more jobs finish?
2. Should the first partial report rank every completed run by matched-strategy router score, or apply the 0.95 final-answer-accuracy threshold from the brief before showing top runs?