continuous_cvae-Apr22-beta-schedule-followup_20260422 Design

Summary

continuous_cvae-Apr22-beta-schedule-followup_20260422 is the direct random-init follow-up to continuous_cvae-Apr1-7loss_20260401. The collection keeps the same six synthetic tasks, the same three adaptation modes, the same single-seed medium model setting, and the same continuous latent dimension 8. The design change is narrower than Apr1: remove the broader auxiliary-loss menu and compare six pure beta schedules while leaving posterior-variance, token-weighted reconstruction, router-support, and router-marginal-KL weights at zero.

Planned design:

• 6 tasks x 6 beta schedules x 3 adaptation modes x 1 seed = 108 jobs
• latent type continuous, continuous_latent_dim=8, last-token pooling
• adaptation modes full, lora, reinit
• optimizer muon, lr=1e-3, 1 epoch, max_sequences_per_epoch=200000
• supervision source groundtruth-data

Relevant repo-side definitions:

• Slurmkit collection: path:.slurmkit/collections/exp2_continuous_cvae-Apr22-beta-schedule-followup.yaml
• Family spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/specs/continuous_cvae_apr22_beta_schedule_followup.yaml
• Parser with canonical schedule labels: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled_random_init/parse_params_continuous_apr22_beta_schedule_followup.py

Why This Collection Exists

Apr1 broadened the continuous-CVAE objective menu to seven settings that mixed normalization, posterior-variance regularization, and token-weighted reconstruction. This Apr22 collection asks a narrower next question before adding new objective families again: if the auxiliary terms stay off, how much of the latent-quality variation can be explained by the schedule of beta alone?

The intended comparison is therefore within the Apr22 schedule family first, and back to Apr1 only as historical context. The design is not trying to retest token-weighted reconstruction or posterior-variance penalties. It is trying to isolate delayed KL onset and cyclical annealing as the main new intervention.

Experiment Metadata

• Raw collection name: exp2_continuous_cvae-Apr22-beta-schedule-followup
• Canonical report name: continuous_cvae-Apr22-beta-schedule-followup_20260422
• Tasks: list_summation, sorting_algorithms, grid_pathfinding, linear_equation_solving, base_conversion, multidigit_addition
• Adaptation modes: full, lora, reinit
• Seed: 314159
• Model-size setting: entangled_model_size=m
• Batch sizes: batch_size=128, eval_batch_size=128
• Evaluation cadence: val_check_interval=0.02
• Runtime instrumentation: profiling_enabled=true

Beta-Schedule Coverage

The six schedule settings play four distinct design roles.

1. beta0_unnorm Zero-beta anchor with reconstruction normalization disabled.
2. beta0p01_d50_lin50 Low-beta delayed-linear schedule with 50% delay and 50% warmup.
3. beta0p1_d50_lin50 Higher-beta delayed-linear schedule with the same 50% delay and 50% warmup.
4. beta1_d50_lin50 High-beta delayed-linear anchor to test whether stronger late KL restores or hurts latent quality.
5. beta0p01_cyclin25_w12p5 Low-beta cyclical-linear schedule with cycle length 25% of training and warmup 12.5%.
6. beta0p1_cyclin25_w12p5 Higher-beta cyclical-linear match to the low-beta cyclical setting.

What stays fixed:

• no token-weighted reconstruction
• no posterior-variance penalty
• no router-support loss
• no router-marginal-KL penalty
• no inter-latent-divergence term
• no multi-seed estimate

Research Questions

1. Which beta schedules best improve router-side disentanglement while preserving final-answer accuracy across the full 6 x 3 task-by-mode grid?
2. At matched beta=0.01 and beta=0.1, how do delayed-linear schedules compare to matched cyclical schedules on router probe, posterior probe, and final accuracy?
3. Are schedule effects consistent across full, lora, and reinit, or does the apparent winner depend on adaptation mode?
4. How does the zero-beta unnormalized anchor compare to the scheduled-beta family on probe quality, accuracy, and strategy-use diagnostics?

Primary Metrics

The main metrics for this collection are:

• sampled_test_final/router_sampled_z/linear_probe_accuracy
• sampled_test_final/final_answer_accuracy
• continuous_diagnostics/test_final/posterior_mu/linear_probe_accuracy
• loss/val/baseline_normalized_reconstruction_loss

The planned deep-dive ranking rule for later follow-up is:

• rank settings by mean sampled_test_final/router_sampled_z/linear_probe_accuracy * sampled_test_final/final_answer_accuracy
• use posterior-mu probe as supporting evidence and tie-break context, not as the primary ranking objective

Secondary metrics to keep visible:

• sampled_test_final/strategy_match_rate
• sampled_test_final/unique_strategy_rate
• sampled_test_final/parse_success_rate
• profile_total/total_seconds

Success Criteria

• all 108 planned cells finish with analyzable artifacts
• the schedule labels remain distinct in the aggregate analysis rather than collapsing into older Mar/Apr settings
• the collection resolves whether delayed-linear or cyclical schedules are the stronger follow-up candidates at low and moderate beta
• the results identify a short deep-dive shortlist without requiring new manual relabeling or ad hoc filtering

Known Risks And Confounders

• beta0_unnorm disables reconstruction normalization, so its stored baseline-normalized reconstruction values reflect unnormalized reconstruction loss
• the collection uses one seed, so mode-level and task-level patterns are directional rather than variance-estimated
• the design keeps all auxiliary losses at zero, so any observed limitation of schedule-only tuning should not be overread as a limit of continuous CVAEs more broadly
• runtime is profiled, but runtime is still a secondary diagnostic here rather than the primary collection goal

Immediate Follow-Up Questions

• If delayed-linear schedules dominate on the agreed composite score, which beta level is the better deep-dive target: the stronger 0.1 schedule or the slightly safer low-beta schedule?
• If cyclical beta=0.01 stays competitive only because it rescues reinit accuracy, is that enough to justify a deeper comparison against the delayed-linear 0.01 setting?
• If zero beta still gives strong router-side probe quality on some tasks but collapses others, should later work treat it as a task-specific probe upper bound rather than a deployable setting?