synthetic-pretrained-disentangled-from-entangled-v3 Design

Summary

synthetic-pretrained-disentangled-from-entangled-v3 is a focused follow-up to synthetic-pretrained-disentangled-from-entangled-v2 and its current partial results. It keeps the entangled-initialized Qwen2.5-0.5B setup and the same 6 synthetic tasks, but narrows the loss sweep to a smaller set of settings that are either the strongest current candidates or the highest-value unresolved tests.

Planned design:

• 6 tasks x 5 loss settings x 1 seed = 30 jobs
• initialization from synthetic-pretrained-entangled-v1
• backbone Qwen2.5-0.5B
• continuous latent, dim 8, last-token pooling

Motivation

The current v2 partial results support three practical updates:

1. Lower effective KL pressure still looks like one of the most plausible ways to improve latent-strategy alignment.
2. Delaying KL pressure appears promising, especially because beta0p1_lin30_pv0 was one of the strongest settings on base_conversion.
3. Mild posterior-variance regularization may help, but v2 does not identify its effect cleanly because the only PV setting is bundled with a different beta schedule and final beta.

At the same time, v2 also surfaced a run-artifact integrity problem caused by local run naming collisions. That should be fixed before the new collection is launched, but it does not change the design question: the next collection should concentrate on a smaller set of higher-value loss settings rather than repeat the broad v2 menu.

Recommended Loss Settings

The proposed collection uses five settings:

1. beta0p02_const_pv0 Current low-beta anchor. This is the strongest constant-beta setting from v2 on several tasks and should remain in the next collection as the main within-family reference.

2. beta0p01_const_pv0 Smaller-beta regime. This directly tests whether the apparent 0.1 -> 0.05 -> 0.02 trend continues when KL pressure is reduced further.

3. beta0p1_lin100_pv0 Longer warmup to the original final beta. This isolates whether the key benefit is primarily about deferring KL pressure, rather than lowering its final value.

4. beta0p05_lin100_pv0 Lower final beta plus full-training warmup, without PV. This is the clean warmup counterpart to the smaller-beta constant settings.

5. beta0p05_lin100_pv0p01 Matched mild-PV comparison. This keeps the same beta and warmup as setting 4, while adding only a small posterior-variance penalty.

All settings keep:

• normalize_reconstruction_loss = true
• router_marginal_kl_to_prior_weight = 0.0
• router_support_weight = 0.0
• token_weighted_reconstruction_weight = 0.0
• inter_latent_divergence_weight = 0.0

Why These Five

This set is intentionally narrow.

• beta0p02_const_pv0 is kept because it is the best current low-beta anchor.
• beta0p01_const_pv0 gives a real smaller-beta test without spending another slot on a more extreme 0.005 regime yet.
• beta0p1_lin100_pv0 is the cleanest timing-only test and directly addresses whether the current beta0p1_lin30_pv0 gains are really about early-training KL pressure.
• beta0p05_lin100_pv0 and beta0p05_lin100_pv0p01 create the matched PV comparison that v2 lacked.

The main omitted setting is beta0p005_const_pv0. That could still be worthwhile later, but the current highest-value uncertainty is not simply “does ever-smaller beta help?” It is “how much of the effect is about final beta versus KL timing, and does mild PV help once timing is already favorable?”

Design Intent

This collection is meant to answer four concrete questions:

1. Does reducing beta from 0.02 to 0.01 continue to improve router-side alignment?
2. Does warming beta from 0 -> 0.1 over the full training run outperform the shorter 30% warmup idea?
3. Is 0.05 with a full warmup a better compromise than the current constant low-beta settings?
4. Does a mild posterior-variance penalty help once beta timing is already favorable, when tested against a matched no-PV control?

Operational Note

This v3 design assumes the run-name / group-name collision issue identified in v2 is fixed before launch. The spec can be authored now, but the collection should not be generated or submitted until the naming fix lands.