from-entangled-v2 continuous-v1 Design

Summary

synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v1 starts a new disentangled-from-entangled lineage rather than extending the older v1 to v5 series. The purpose of the new line is to rerun the most decision-relevant continuous-latent loss settings against the newer pretrained entangled source family, including a clean token-weighted reconstruction comparison under the current teacher / tokenizer / layout setup.

Planned design:

• 6 tasks x 6 loss settings x 1 model x 1 seed = 36 jobs
• initialization from the entangled source collection synthetic-pretrained-entangled-v2-tokenization-layout
• backbone Qwen2.5-0.5B only in this initial collection
• continuous latent, dim 8, last-token pooling, lora adaptation

Relevant repo-side definitions:

• slurmkit collection: path:.slurmkit/collections/synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v1.yaml
• job spec: path:experiments/synthetic_sequences/slurm/families/disentangled_from_entangled/specs/continuous_from_entangled_v2_v1_qwen25_05b.yaml

Why This Is A New Lineage

The earlier entangled-initialized pretrained sweeps culminated in synthetic-pretrained-disentangled-from-entangled-v5-twr, which identified a small set of promising low-KL and TWR settings but still depended on the older entangled source collection. This from-entangled-v2 line exists to answer the same core disentangling questions against the newer entangled source family, so that any TWR conclusions are grounded in the currently supported pretrained source setup rather than inherited from an older teacher line.

This note therefore treats the older synthetic-pretrained-disentangled-from-entangled-v1 through v5-twr sequence as the main design predecessor, but not as the same collection family. Comparisons across the two lines should be interpreted as cross-line replication or robustness checks, not as simple within-line continuations.

Experiment Metadata

• Collection: synthetic-pretrained/disentanglement/from-entangled-v2/continuous-v1
• Trainer collection name: synthetic-pretrained-disentangled-from-entangled-v2-continuous-v1
• Source collection name for teacher lookup: synthetic-pretrained-entangled-v2-tokenization-layout
• Supervision source: groundtruth-data
• Seed: 314159
• Training budget: 1 epoch, max_sequences_per_epoch=200000
• Batching: batch_size=128, micro_batch_size=auto, eval_batch_size=128
• Auto micro-batch threshold: 0.9
• Optimization: adamw, lr=2e-4, linear-warmup cosine decay, weight_decay=0.01
• Runtime acceleration: amp_bfloat16=true, torch_compile=true
• Historical drafting snapshot: this design note was first drafted when 32/36 jobs were complete and 4/36 were still running. The collection is now complete; see results for the refreshed quantitative snapshot.

Experimental Coverage

Tasks

The collection spans the full six-task synthetic suite:

• list_summation
• sorting_algorithms
• grid_pathfinding
• linear_equation_solving
• base_conversion
• multidigit_addition

Loss settings

The sweep intentionally keeps only six settings:

1. beta1_const_pv0_unnorm Standard ELBO anchor with unnormalized reconstruction loss.
2. beta0_const_pv0 Zero-beta reconstruction anchor.
3. beta0p005_const_pv0 Strongest constant low-beta no-TWR carryover.
4. beta0p01_lin100_pv0 Strongest full-warmup no-TWR carryover.
5. beta0p005_const_pv0_twr0p5 Strongest constant-beta TWR carryover.
6. beta0p01_lin100_pv0_twr0p5 Strongest warmup TWR carryover.

This gives the collection four distinct design roles:

• one true ELBO non-identifying baseline
• one zero-beta anchor
• two strongest non-TWR candidates from the earlier line
• two strongest TWR candidates from the earlier line

Posterior-variance settings are deliberately omitted. The earlier from-entangled line did not justify spending new collection budget on PV in preference to the cleaner ELBO / low-beta / TWR contrasts.

What is not covered

• no qwen25_15b runs in this initial collection
• no posterior-variance sweep
• no router-marginal-KL or inter-latent-divergence sweep
• no direct-pretrained arm inside the same collection
• no multi-seed estimate of variance

Research Questions

This collection is meant to answer six concrete questions.

1. Does the entangled-initialized continuous factorization still produce useful latent structure when trained against the newer pretrained entangled source family?
2. Relative to the standard ELBO anchor, do the low-KL settings again move runs away from the collapsed regime in which the factorized model simply matches the source model while leaving z inert?
3. Does the zero-beta anchor remain competitive, or do small nonzero-beta settings still produce a better reconstruction-vs-structure tradeoff?
4. Do the two strongest TWR settings outperform their matched no-TWR predecessors once tested against the new source lineage?
5. Is the apparent TWR benefit, if present, visible on router-side metrics, posterior-side metrics, or both?
6. Are the strongest findings from the earlier from-entangled line reproducible under the new source setup, or were they partly contingent on the older teacher family?

Paper-Facing Role

This collection is primarily relevant to the claims and figure notes around disentanglement quality, reconstruction pressure, and semantic stability:

• Paper Claims, especially the current draft claims C2, C3, C4, and C7
• Figure 3 — latent-space disentanglement across training
• Figure 4 — base-normalized reconstruction vs. strategy decodability
• Figure 5 — initialization pathology
• Body-figure candidate — cross-input semantic stability / analogical agreement

The main expected paper contribution from this collection is not a new objective family, but a cleaner pretrained-source validation of the current best candidate settings. If the strongest no-TWR and TWR regimes replicate here, that strengthens the case that the previously observed reconstruction-vs-structure frontier is not an artifact of one older entangled teacher line.

Methodological Dependencies

This design depends directly on:

• Distribution-Level Z-Free-Relative Reconstruction Rescaling
• Token-level surprisal-share weighting
• Synthetic-sequences disentangled factorization

Methodologically, the collection is asking whether the current reconstruction-pressure interventions remain productive when the initialization source is changed, not whether those interventions should be rederived from scratch.

Primary Metrics

The main metrics to emphasize in analysis are:

• posterior probe accuracy / strategy decodability
• router probe accuracy / strategy decodability
• loss/val/baseline_normalized_reconstruction_loss
• sampled validation or test final-answer accuracy
• sampled validation or test strategy-alignment metrics
• analogical-agreement or cross-input semantic-stability diagnostics, if the current analysis pipeline exposes them cleanly for this line

The core decision axis should remain the same as in the paper notes: whether lower relative reconstruction ratio and stronger strategy decodability move together, and which loss settings sit on the favorable part of that frontier.

Secondary Metrics And Comparisons

Useful secondary summaries include:

• per-task winner tables rather than only global means
• matched TWR-vs-no-TWR deltas against the older v4 and v5-twr no-TWR baselines
• within-line ELBO-vs-low-beta-vs-TWR comparisons
• comparison against the direct-pretrained disentangling line where relevant to the initialization-pathology question

Because this is a new source-model lineage, the first comparison priority should be within this collection, followed by careful cross-line comparisons to the earlier synthetic-pretrained-disentangled-from-entangled-v1 through v5-twr series.

Success Criteria

• all 36 planned cells finish with analyzable artifacts
• at least one non-ELBO setting clearly improves over the ELBO anchor on latent-structure metrics without destroying task accuracy
• the strongest earlier low-beta regimes remain competitive under the new source setup
• the TWR settings are evaluated cleanly enough to decide whether they deserve to remain in the paper-facing shortlist
• the collection yields a small, defensible set of settings to carry forward into future larger-model or multi-seed follow-ups

Known Risks And Confounders

• sorting_algorithms is the main operational risk. It already required auto micro-batch recovery on this collection, so future interpretation should distinguish optimizer behavior from pure memory failures.
• The collection currently uses only one pretrained backbone and one seed, so any apparent cross-task or cross-setting pattern should be treated as directional rather than fully stable.
• Cross-line comparisons to the older from-entangled v1 to v5 sequence are meaningful, but they are not same-line ablations because the source teacher family changed.
• TWR settings are computationally heavier than no-TWR settings, so any remaining completion asymmetry can bias naive aggregate comparisons if unfinished or retried runs are handled carelessly.

Immediate Follow-Up Questions

• If one TWR setting dominates, should the next from-entangled-v2 follow-up widen the model axis (qwen25_15b) before widening the loss menu again?
• If the TWR advantage does not replicate, should the new line pivot back toward pure low-KL sweeps rather than carrying TWR forward as a headline intervention?
• If the new source line improves semantic-stability diagnostics more than raw probe accuracy, should later reports emphasize cross-input consistency more centrally in the narrative?