from-entangled-v2 continuous-v3 beta-schedule Design

from-entangled-v2 continuous-v3 beta-schedule Design

Summary

synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v3-beta-schedule is the schedule-only follow-up to from-entangled-v2 continuous-v2. The collection keeps the same six synthetic tasks, the same entangled-v2 teacher lineage, the same single-seed Qwen2.5-0.5B setting, the same continuous latent dimension 8, and the same lora adaptation mode. The design change is narrow and explicit: replace the older six-setting objective shortlist from continuous-v2 with the six beta schedules approved by the Apr22 random-init follow-up.

Planned design:

  • 6 tasks x 6 beta schedules x 1 mode x 1 seed = 36 jobs
  • source teachers from synthetic-pretrained-entangled-v2-tokenization-layout
  • continuous latent, continuous_latent_dim=8, last-token pooling, lora adaptation
  • 1 epoch, max_sequences_per_epoch=200000, batch_size=128, eval_batch_size=128

Relevant repo-side definitions:

  • slurmkit collection: path:.slurmkit/collections/synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v3-beta-schedule.yaml
  • family spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled/specs/continuous_from_entangled_v2_v3_beta_schedules_qwen25_05b.yaml
  • parser with canonical schedule labels: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled/params_from_entangled_v2_v2.py

Why This Collection Exists

continuous-v2 showed that the entangled-v2-source lineage could sustain strong router-side and posterior-side decodability, but that collection still mixed schedule decisions with the older low-beta and token-weighted-reconstruction objective shortlist. The Apr22 random-init follow-up narrowed the schedule search to six pure beta schedules and produced a clean shortlist centered on delayed-linear and cyclical variants.

This collection asks the next concrete transfer question: if the teacher family and the pretrained LoRA defaults stay fixed, do those same six schedule-only settings remain competitive when imported into the entangled-v2-source continuous rerun? The intended comparison is therefore within the new schedule family first, with continuous-v2 serving as the immediate design predecessor and Apr22 serving as the schedule-menu source.

Experiment Metadata

  • Slurmkit collection: synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v3-beta-schedule
  • Raw trainer collection name: synthetic-pretrained-disentangled-from-entangled-v2-continuous-v2-beta-schedule-followup
  • Teacher lookup collection: synthetic-pretrained-entangled-v2-tokenization-layout
  • Supervision source: groundtruth-data
  • Seed: 314159
  • Pretrained backbone: entangled-source Qwen2.5-0.5B
  • Latent configuration: continuous, continuous_latent_dim=8, pooling=last_token
  • Adaptation mode: lora
  • Optimization: adamw, lr=2e-4, linear-warmup cosine decay, weight_decay=0.01
  • Runtime defaults: amp_bfloat16=true, torch_compile=true

Beta-Schedule Coverage

The six settings play four distinct roles:

  1. beta0_unnorm Zero-beta anchor with reconstruction normalization disabled.
  2. beta0p01_d50_lin50 Low-beta delayed-linear schedule with 50% delay and 50% warmup.
  3. beta0p1_d50_lin50 Moderate-beta delayed-linear schedule with the same 50% delay and 50% warmup.
  4. beta1_d50_lin50 Higher-beta delayed-linear anchor.
  5. beta0p01_cyclin25_w12p5 Low-beta cyclical-linear schedule with cycle length 25% of training and warmup 12.5%.
  6. beta0p1_cyclin25_w12p5 Moderate-beta cyclical-linear match to the low-beta cyclical setting.

What stays fixed:

  • no token-weighted reconstruction
  • no posterior-variance penalty
  • no router-support loss
  • no router-marginal-KL penalty
  • no inter-latent-divergence term
  • no multi-seed estimate

Research Questions

  1. Which beta schedule settings best improve router-side disentanglement while preserving final-answer accuracy in the entangled-v2-source LoRA setting?
  2. At matched beta=0.01 and beta=0.1, how do delayed-linear schedules compare to matched cyclical schedules on router probe, posterior probe, reconstruction, and the agreed router probe x final answer accuracy composite?
  3. How competitive is the zero-beta unnormalized anchor relative to the scheduled-beta family once answer-accuracy drops are included in the ranking?
  4. Which task-specific exceptions are large enough to justify a deeper follow-up rather than a collection-level conclusion?

Falsifiable Hypotheses

  • The delayed-linear schedules will outrank the matched cyclical schedules on the agreed composite metric at the same beta.
  • beta0_unnorm will remain strong on raw router probe on at least some tasks, but its accuracy drop will keep it from being the strongest global setting.
  • The cyclical beta=0.1 setting will be weaker than the delayed beta=0.1 setting on router probe and posterior probe.

Primary Metrics

The main metrics for this collection are:

  • sampled_test_final/router_sampled_z/linear_probe_accuracy
  • sampled_test_final/final_answer_accuracy
  • continuous_diagnostics/test_final/posterior_mu/linear_probe_accuracy
  • loss/val/baseline_normalized_reconstruction_loss

The agreed headline ranking rule is:

  • rank settings by mean sampled_test_final/router_sampled_z/linear_probe_accuracy * sampled_test_final/final_answer_accuracy
  • treat raw router probe as supporting evidence rather than the only objective

Secondary metrics to keep visible:

  • sampled_test_final/strategy_match_rate
  • sampled_test_final/unique_strategy_rate
  • sampled_test_final/inferred_strategy_coverage_rate
  • sampled_test_final/off_strategy_rate
  • sampled_test_final/parse_success_rate

Success Criteria

  • all 36 planned cells finish with analyzable artifacts
  • the collection resolves whether the imported delayed-linear schedules remain stronger than the matched cyclical schedules in this pretrained LoRA setting
  • the results identify a short follow-up shortlist without requiring ad hoc relabeling or manual filtering
  • task-level exceptions are explicit enough to decide whether a later deep dive is needed

Known Risks And Confounders

  • beta0_unnorm disables reconstruction normalization, so its stored baseline-normalized reconstruction values are descriptive only
  • the design uses one seed and one adaptation mode, so the observed schedule ranking is directional rather than variance-estimated
  • the collection imports the Apr22 schedule menu but not the Apr22 random-init optimization or mode grid, so those earlier results are layout/context references rather than matched numerical baselines inside results.md
  • this collection narrows the objective family intentionally; any weak setting here is evidence about this schedule menu, not about all continuous disentangling objectives
Built with LogoFlowershow