continuous_cvae-Apr26-reconstruction-stack-followup_20260426 Design

Back to hub

Summary

continuous_cvae-Apr26-reconstruction-stack-followup_20260426 is a continuous-CVAE follow-up to continuous_cvae-Apr24-reconstruction-stack_20260424. It keeps the Apr24 task grid, adaptation modes, latent dimension, optimizer, one-epoch training budget, and single seed, but narrows the reconstruction menu to the cells needed to resolve the main Apr24 ambiguities.

Planned design:

• 6 tasks x 9 loss settings x 3 adaptation modes x 1 seed = 162 jobs
• seed 314159
• latent type continuous, continuous_latent_dim=8, last-token pooling
• adaptation modes full, lora, reinit
• optimizer muon, lr=1e-3, 1 epoch, max_sequences_per_epoch=200000
• supervision source groundtruth-data
• reconstruction-stack follow-up settings from path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_followup_sweep.py

Relevant repo-side definitions:

• Slurmkit collection: path:.slurmkit/collections/synthetic_sequences/families/disentangled_from_entangled_random_init/continuous_cvae_apr26_reconstruction_stack_followup.yaml
• Family spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/specs/continuous_cvae_apr26_reconstruction_stack_followup.yaml
• Parser: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/parse_params_continuous_apr26_reconstruction_stack_followup.py
• Shared reconstruction menu: path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_followup_sweep.py

Why This Collection Exists

Apr24 showed that normalized reconstruction variants were more promising than the raw beta-1 ELBO-style baseline on the router-side strategy frontier, with distribution-normalized plus normalized concentration-corrected token-weighted reconstruction among the strongest settings. It also left several important comparisons unresolved: whether cc-TWR works without a distribution-level term, whether example-level inverse-surprisal can replace the distribution-level term in a pairwise stack, whether raw reconstruction is weak because it is raw or because the Apr24 raw baseline used beta 1, and whether the Apr22 beta 0.1 delayed-linear signal still helps the best Apr24 normalized family.

This Apr26 collection is therefore not a broad exploratory sweep. It is a gap-closing collection designed to isolate the most decision-relevant Apr24 confounds before choosing the next reconstruction objective for larger or replicated runs.

Sweep Coverage

Tasks:

• base_conversion
• grid_pathfinding
• linear_equation_solving
• list_summation
• multidigit_addition
• sorting_algorithms

Adaptation modes:

• full
• lora
• reinit

Loss settings:

The collection deliberately omits the Apr24 distribution-normalized plus cc-TWR beta 0.01 cells because those are inherited from the predecessor collection as the main comparison points. It also omits alpha 1.0 cc-TWR-only settings because Apr24 already suggested that alpha 0.5 was the more relevant continuation target for the strongest distribution-plus-token family.

Research Questions

1. Does raw reconstruction remain weak when beta is reduced to 0.01, or was the Apr24 raw baseline mainly harmed by the standard ELBO beta scale?
2. Can cc-TWR alone recover router-side strategy information at acceptable final-answer accuracy, or does it require a distribution-level or example-level companion term?
3. Within cc-TWR-only settings, does token alpha 0.25 improve stability relative to alpha 0.5, or does the weaker concentration pressure lose strategy signal?
4. Does example inverse-surprisal plus cc-TWR match or improve on distribution-normalized plus cc-TWR, especially on tasks where high-surprisal examples may dominate raw loss?
5. Does the beta 0.1 delayed-linear continuation of distribution-normalized plus cc-TWR improve over the Apr24 beta 0.01 setting, or does the stronger KL pressure hurt router-side usability?
6. Do constant and delayed-linear schedules have the same direction of effect for raw, exinv+cc-TWR, and cc-TWR-only families?
7. Are the Apr26 objective effects consistent across full, lora, and reinit, or do the adaptation modes prefer different reconstruction stacks?

Falsifiable Hypotheses

H1. Low-beta raw reconstruction will improve over Apr24 raw beta-1 baselines, but will still underperform normalized reconstruction variants on analysis/router_probe_times_strategy_match_rate at comparable final-answer accuracy.

H2. cc-TWR-only settings will recover some router-side strategy signal, but the best pairwise stacks will remain stronger if distribution-level or example-level normalization supplies a broader reconstruction scale.

H3. Token alpha 0.25 will be more stable but less discriminative than alpha 0.5; if alpha 0.25 wins, the Apr24 token concentration pressure was likely too sharp for standalone use.

H4. Example inverse-surprisal plus cc-TWR will help on tasks where rare or difficult examples carry useful strategy information, but may trade away average final-answer accuracy if it underweights common easy examples too aggressively.

H5. The beta 0.1 delayed-linear distribution-plus-token continuation will either improve the Apr24 beta 0.01 frontier or reveal that the Apr22 beta 0.1 advantage was schedule-family-specific rather than objective-family-general.

H6. Delayed-linear timing will remain useful when paired with cc-TWR, but raw low-beta reconstruction may show a smaller schedule effect because its reconstruction target is not normalized.

Primary Metrics

• analysis/router_probe_times_strategy_match_rate: headline ranking metric, following the Apr22 and Apr24 "matched-strategy router score" convention.
• sampled_test_final/router_sampled_z/linear_probe_accuracy: primary router-side latent strategy decodability.
• sampled_test_final/final_answer_accuracy: primary task-performance guardrail.
• sampled_test_final/strategy_match_rate: behavioral strategy-use guardrail for the headline composite.
• continuous_diagnostics/test_final/posterior_mu/linear_probe_accuracy: posterior-side decodability, used to separate inference-side structure from router-side usability.
• loss/val/baseline_normalized_reconstruction_loss or final-eval equivalent when available: base-relative reconstruction ratio for reconstruction-decodability analysis.

Secondary metrics:

• sampled_test_final/unique_strategy_rate
• sampled_test_final/parse_success_rate
• sampled_test_final/invalid_structure_rate
• sampled_test_final/router_variance_mean
• loss/test/reconstruction_loss
• loss/test/distribution_normalized_reconstruction_loss
• loss/test/example_inverse_surprisal_weighted_reconstruction_loss
• loss/test/normalized_concentration_corrected_token_weighted_reconstruction_loss
• loss/test/kl_prior_loss
• profile_total/total_seconds

Figure Plan

F0. Completeness and run-identity audit. Heatmap of expected task x loss_setting x adaptation_mode cells with statuses for complete, running, pending, failed, missing artifact, duplicate artifact, and unmatched artifact. This is the gate figure for any partial rerun.

F1. Apr26 frontier summary. Dot or interval plots for final accuracy, router probe, strategy match rate, posterior probe, baseline-normalized reconstruction, and analysis/router_probe_times_strategy_match_rate by loss setting and adaptation mode.

F2. Matched Apr24-to-Apr26 contrast table. A compact contrast view that imports Apr24 comparison anchors for dist_plus_tokcc_a0p5_b0p01_const, dist_plus_tokcc_a0p5_b0p01_d50_lin50, and raw beta-1 rows, then compares them against Apr26 raw beta 0.01, cc-TWR-only, exinv+cc-TWR, and dist+cc-TWR beta 0.1.

F3. cc-TWR-only alpha comparison. Matched deltas for alpha 0.25 versus alpha 0.5 under constant and delayed-linear schedules, faceted by adaptation mode and optionally task.

F4. Schedule comparison within Apr26 families. Delayed-linear minus constant deltas for raw beta 0.01, exinv+cc-TWR, cc-TWR alpha 0.25, and cc-TWR alpha 0.5, with final-answer accuracy shown alongside the matched-strategy router score.

F5. Pairwise-normalizer comparison. exinv+cc-TWR versus Apr24 dist+cc-TWR beta 0.01 at alpha 0.5, using matched task and adaptation-mode deltas where possible.

F6. Beta continuation comparison. Apr26 dist_plus_tokcc_a0p5_b0p1_d50_lin50 versus Apr24 dist_plus_tokcc_a0p5_b0p01_d50_lin50, with special attention to whether the stronger beta improves strategy match, router probe, or only posterior-side structure.

F7. Reconstruction ratio versus decodability scatter. One point per valid run, with base-normalized reconstruction ratio on the x-axis and router probe or matched-strategy router score on the y-axis, colored by reconstruction family and shaped by adaptation mode.

F8. Training dynamics for diagnostic settings. Validation curves for final accuracy, router probe, posterior probe, reconstruction ratio, component losses, and KL for top composite runs, cc-TWR-only winners or failures, low-beta raw controls, and the beta 0.1 dist+cc-TWR continuation.

Analysis Plan And Missing-Run Handling

The repo-side analysis should be idempotent and partial-grid aware:

1. Build expected_cells.csv from the slurmkit collection jobs, with one row per planned (task_name, loss_setting, adaptation_mode, seed) cell plus all resolved reconstruction weights, beta schedule fields, job ids, and latest slurm state.
2. Discover run artifacts under path:.runs/synthetic_sequences/disentangled/exp2_continuous_cvae-Apr26-reconstruction-stack-followup, but do not use directory count as the denominator. A run is analyzable only when run_manifest.json and metrics.json exist and its manifest fields match exactly one expected cell.
3. Derive missing loss_setting, loss_setting_family, and beta_schedule_setting values from exact reconstruction-weight and beta-schedule signatures, using LOSS_SETTINGS from path:experiments/synthetic_sequences/slurm/families/shared/reconstruction_stack_followup_sweep.py.
4. Left-join observed artifacts onto expected cells and assign a status column: complete, running, pending, failed, missing_artifact, duplicate_artifact, or unmatched_artifact. Scientific figures should filter to complete; completeness figures should show all statuses.
5. Generate aggregate summaries over available completed runs, always reporting n_complete and n_expected for each group. Any mean, delta, or ranking must carry its denominator so partial snapshots are not mistaken for complete-grid evidence.
6. Save outputs under path:experiments/synthetic_sequences/disentangled/analysis/aggregates/continuous_cvae-Apr26-reconstruction-stack-followup_20260426, including summary.csv, metrics_long.csv, eval_history_long.csv, expected_cells.csv, missing_or_incomplete_cells.csv, unmatched_artifacts.csv, aggregate_manifest.json, findings.json, and report.md.
7. Re-running the analysis after additional jobs finish should require only aggregate and figure regeneration; no hand-maintained missing-run list should be needed.

Empirical Claims Requiring Evidence

1. Low-beta raw reconstruction is or is not a fairer raw baseline than Apr24 beta-1 raw reconstruction. Evidence: raw beta 0.01 versus Apr24 raw beta 1 contrasts, plus final-answer accuracy and KL.
2. cc-TWR alone is sufficient or insufficient as a reconstruction objective for router-side strategy recovery. Evidence: cc-TWR-only alpha 0.25 and alpha 0.5 rows versus Apr24 dist+cc-TWR anchors.
3. Token alpha has a measurable standalone effect. Evidence: matched alpha 0.25 versus alpha 0.5 deltas at fixed task, adaptation mode, beta, and schedule.
4. Example inverse-surprisal can or cannot replace distribution-level normalization in a pairwise stack with cc-TWR. Evidence: exinv+cc-TWR versus dist+cc-TWR anchors, with task-level deltas.
5. The beta 0.1 delayed-linear continuation improves or worsens the current best dist+cc-TWR family. Evidence: Apr26 beta 0.1 dist+cc-TWR versus Apr24 beta 0.01 dist+cc-TWR delayed-linear.
6. Schedule effects are family-dependent or stable across Apr26 objectives. Evidence: delayed-linear minus constant deltas for every Apr26 family with both schedules.
7. Adaptation modes respond differently to the follow-up objectives. Evidence: per-mode frontier plots and matched-mode deltas for full, lora, and reinit.
8. Any paper-facing claim about reconstruction normalization, cc-TWR, or raw ELBO failure is restricted to complete, uniquely matched artifacts and reports denominators for partial snapshots. Evidence: completeness audit, aggregate manifest, and unmatched-artifact table.

Success Criteria

• Every planned cell has a unique, auditable mapping from intended loss_setting to run artifact or an explicit incomplete status.
• The aggregate can recover all Apr26 reconstruction-weight fields from trusted sources and preserve them as separate columns.
• The headline ranking uses analysis/router_probe_times_strategy_match_rate, while final-answer accuracy remains visible as a guardrail.
• The analysis can decide whether the next collection should continue with dist+cc-TWR, exinv+cc-TWR, cc-TWR-only, or raw low-beta controls.
• The figure set can directly answer whether the Apr24 strongest setting was strong because of distribution normalization, token concentration, alpha, beta schedule, or their interaction.

Known Risks And Confounders

• The collection uses one seed, so task- and mode-level patterns are directional rather than variance-estimated.
• Several key contrasts compare Apr26 cells against Apr24 anchors rather than within the same Apr26 spec. Those comparisons should use matched tasks, adaptation modes, and seed, and should keep collection provenance visible.
• The beta 0.1 distribution-plus-token row has no constant-schedule counterpart in Apr26, so it tests a targeted continuation rather than a full beta-by-schedule grid.
• cc-TWR-only rows isolate token concentration, but they may change component-loss scales enough that raw component values and weighted contributions must be shown before interpreting quality differences.
• If loss_setting is not persisted directly in run manifests, the analysis must verify derived settings against explicit reconstruction weights, beta fields, and loss_tag.
• Runtime is secondary; profiling can explain analysis cost and objective overhead but should not drive the scientific ranking unless quality metrics are tied.