from-entangled-v2 continuous-v2 Design

Summary

synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v2 is the direct matched follow-up to from-entangled-v2 continuous-v1. The collection keeps the same six tasks, the same six curated loss settings, the same pretrained entangled teacher lineage, and the same single-seed Qwen2.5-0.5B setup. The intended modeling change is narrow and explicit: increase continuous_latent_dim from 8 to 64.

Planned design:

• 6 tasks x 6 loss settings x 1 model x 1 seed = 36 jobs
• initialization from synthetic-pretrained-entangled-v2-tokenization-layout
• backbone Qwen2.5-0.5B only
• continuous latent, dim 64, last-token pooling, lora adaptation

Relevant repo-side definitions:

• slurmkit collection: path:.slurmkit/collections/synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v2.yaml
• job spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled/specs/continuous_from_entangled_v2_v2_qwen25_05b.yaml

Why This Collection Exists

continuous-v1 established the first entangled-v2-source continuous rerun with continuous_latent_dim=8. This continuous-v2 collection exists to answer the next direct question before broadening the design again: if the teacher family and loss menu stay fixed, does a much larger continuous latent improve router-side decodability, posterior-side decodability, reconstruction, or semantic alignment?

This is therefore a replication-plus-axis-change collection, not a new sweep family. The correct comparison target is the matched continuous-v1 collection, not the older synthetic-pretrained-disentangled-from-entangled-v1 through v5-twr line except as historical context.

Experiment Metadata

• Collection: synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v2
• Trainer collection name: synthetic-pretrained-disentangled-from-entangled-v2-continuous-v2
• Direct reference collection: synthetic-pretrained-disentangled-from-entangled-v2-continuous-v1
• Source collection name for teacher lookup: synthetic-pretrained-entangled-v2-tokenization-layout
• Supervision source: groundtruth-data
• Seed: 314159
• Training budget: 1 epoch, max_sequences_per_epoch=200000
• Batching: batch_size=128, micro_batch_size=auto, eval_batch_size=128
• Auto micro-batch threshold: 0.9
• Optimization: adamw, lr=2e-4, linear-warmup cosine decay, weight_decay=0.01
• Runtime acceleration: amp_bfloat16=true, torch_compile=true
• Current execution snapshot while this design note is being drafted: 33/36 complete, 3/36 still running

Matched Change Versus continuous-v1

Primary intended change:

• continuous_latent_dim: 8 -> 64

Regenerated runtime-spec drift that should be documented but not treated as the primary interpretation target:

• micro_batch_size: null/implicit -> auto
• auto_micro_batch_max_memory_fraction: absent -> 0.9
• profiling_enabled: false -> true

The design goal is still to treat latent dimensionality as the main changed axis. The runtime drift is relevant context for caution, not the headline experimental variable.

Experimental Coverage

Tasks

The collection keeps the same six-task synthetic suite:

• list_summation
• sorting_algorithms
• grid_pathfinding
• linear_equation_solving
• base_conversion
• multidigit_addition

Loss settings

The loss menu is identical to continuous-v1:

1. beta1_const_pv0_unnorm
2. beta0_const_pv0
3. beta0p005_const_pv0
4. beta0p01_lin100_pv0
5. beta0p005_const_pv0_twr0p5
6. beta0p01_lin100_pv0_twr0p5

This preserves the same four design roles:

• one ELBO anchor
• one zero-beta anchor
• two no-TWR low-KL candidates
• two matched TWR candidates

What is not covered

• no model-size broadening
• no extra loss families beyond the continuous-v1 shortlist
• no multi-seed variance estimate
• no direct-pretrained arm inside this collection
• no posterior-variance or router-marginal-KL expansion

Research Questions

This collection is meant to answer five concrete questions.

1. Against the newer entangled-v2 teacher line, which of the six curated loss settings still move the factorized model away from the ELBO-collapsed corner while preserving high task accuracy?
2. Do the matched TWR settings still improve the reconstruction-vs-disentanglement frontier relative to their no-TWR baselines inside this collection?
3. Are the strongest gains visible on router-sampled metrics, posterior-mu metrics, or both?
4. Relative to continuous-v1, what is the effect of increasing continuous_latent_dim from 8 to 64 on router probe, posterior probe, reconstruction, and final-answer accuracy?
5. Is any latent-dimension effect consistent across tasks, or does it interact strongly with task and loss setting?

Paper-Facing Role

This collection is most relevant to the same paper-facing claims and figure families as continuous-v1, but now with an explicit latent-dimension robustness angle:

• Paper Claims, especially the current draft claims C2, C3, C4, and C7
• Figure 3 — latent-space disentanglement across training
• Figure 4 — base-normalized reconstruction vs. strategy decodability
• Figure 5 — initialization pathology
• Body-figure candidate — cross-input semantic stability / analogical agreement

The expected contribution is not a new objective family. It is a matched answer to whether a much larger continuous latent helps or hurts the current best candidate regimes.

Primary Metrics

The main metrics to emphasize in analysis remain:

• sampled-test router probe accuracy
• test posterior-mu probe accuracy
• loss/val/baseline_normalized_reconstruction_loss
• sampled-test final-answer accuracy
• sampled-test strategy-match and uniqueness summaries

For the cross-dimension comparison, the main deltas of interest are:

• dim64 - dim8 router probe
• dim64 - dim8 posterior probe
• dim64 - dim8 baseline-normalized reconstruction
• dim64 - dim8 final-answer accuracy

Success Criteria

• all 36 planned cells finish with analyzable artifacts
• the collection cleanly resolves whether increasing latent dimensionality improves any of the currently paper-relevant regimes
• the matched dim-8 vs dim-64 comparison is precise enough to say which loss settings benefit, regress, or remain unchanged
• the report either supports carrying high-dimensional latents forward or gives a concrete reason not to

Known Risks And Confounders

• The three unfinished cells are concentrated in sorting_algorithms TWR settings and base_conversion / beta0_const_pv0, so current partial comparisons are not uniformly complete across the design.
• The regenerated runtime spec adds non-primary differences in micro-batching and profiling, so any subtle effect should not be overclaimed as purely dimensional.
• The collection still uses one seed and one backbone, so cross-task or cross-setting patterns are directional rather than fully stable.
• The beta1_const_pv0_unnorm setting still carries the known normalized-reconstruction caveat: when baseline normalization is disabled, the stored normalized-reconstruction field is not directly comparable to the normalized settings.

Immediate Follow-Up Questions

• If the larger latent mostly hurts the low-KL and TWR regimes, should future work return to smaller continuous latents rather than scaling them up further?
• If the ELBO anchor is the only clear beneficiary of higher latent dimension, does that reflect genuine extra capacity or a different collapse-adjacent regime?
• If one task family benefits while others regress, should later follow-ups tune latent dimension by task or by objective family rather than trying to choose one global value?