Qwen25 TOK-CC Hyperparameter Follow-Up: Design

Back to hub

Motivation

The parent Qwen2.5 multi-task paper-method grid showed an unusual 1.5B pattern: pure tokcc_a0p25_b0p01_lin100 had weak strong-strategy recovery, while Qwen2.5 0.5B and the earlier random-init experiments did not show the same failure mode. The pattern is especially suspicious because tokcc_a0p25_b0p1_lin100 was strong on 1.5B, and because the small-mixture row dist0p05_tokcc0p95_a0p25_b0p01_lin100 was strong on 1.5B despite sharing the same alpha and beta as the weak pure TOK-CC row.

This follow-up is designed to separate four explanations: random seed instability, genuine beta sensitivity, alpha sensitivity, and stabilization from adding a small distribution-normalized reconstruction term.

Research Questions

1. Does rerunning pure tokcc_a0p25 at new downstream disentangled seeds reproduce the weak 1.5B b0p01 result, or was the original cell an unlucky run?
2. Is the 1.5B contrast between b0p01 and b0p1 stable under seed replication?
3. Do pure TOK-CC alpha endpoints 0.1 and 0.5 improve over the existing 0.25 setting for either model size or beta?
4. Does a small distribution-normalized component, dist0p05_tokcc0p95, stabilize TOK-CC across alpha endpoints and seeds?
5. Are the answer-accuracy and parse-success guardrails preserved while improving router-side strategy recovery?

Sweep

Fixed settings:

• Task: multi_task
• Source collection: synthetic-pretrained-entangled-v2-tokenization-layout-multi-task-qwen25
• Source entangled seed: 314159
• Model keys: qwen25_05b, qwen25_15b
• Continuous latent dimension: 64
• Adaptation: LoRA
• Epochs: 1
• Batch size: 128
• Eval batch size: 16
• Beta schedule: lin100
• Same-job analogical steering: enabled with num_pairs=1024, batch_size=1, skip_existing=true

Planned cells:

• Pure TOK-CC alpha endpoints: tokcc_a0p1, tokcc_a0p5 crossed with model key and beta b0p01, b0p1 at downstream seed 314159 (8 jobs).
• Pure TOK-CC alpha-0.25 seed replicates: tokcc_a0p25 crossed with model key and beta b0p01, b0p1 at downstream seeds 271828, 161803 (8 jobs).
• Small-dist stabilizer alpha endpoints: dist0p05_tokcc0p95_a0p1, dist0p05_tokcc0p95_a0p5 crossed with model key and beta b0p01, b0p1 at downstream seed 314159 (8 jobs).
• Small-dist stabilizer seed replicates for the key 1.5B b0p01 row: dist0p05_tokcc0p95_a0p25_b0p01_lin100 at downstream seeds 271828, 161803 (2 jobs).

The 12/14 model split is intentional. The shared core design contributes 12 cells per model; the extra two 1.5B cells replicate the small-dist stabilizer row most directly implicated by the parent paper-method anomaly.

Submission Layout

The 26 jobs are split across two slurmkit collections to reduce queue coupling:

• tokcc-hparam-followup-multi-task-z64-qwen25-lafferty-h100: 13 jobs on the lafferty H100 reservation.
• tokcc-hparam-followup-multi-task-z64-qwen25-general: 13 jobs on the general GPU cluster with h200|h100|a100 constraint.

Both slurmkit collections write to the same training-run collection name, synthetic-pretrained-disentangled-from-entangled-v2-tokcc-hparam-followup-multi-task-z64-qwen25, so later aggregate analysis can treat them as one 26-cell experiment.

Metrics

Primary metrics are matched router score, analogical set overlap, and within-task router probe accuracy. Guardrail metrics are final answer accuracy, parse success, sampled strategy-match rate, all-task router probe accuracy, posterior-mu probe accuracy, reconstruction losses, and KL-to-prior loss.

For seed-replicate questions, compare the new downstream seeds against the existing seed-314159 paper-method grid cells rather than treating this collection alone as a complete three-seed sweep.

Expected Readouts

• If pure tokcc_a0p25_b0p01 stays weak across 1.5B seed replicates but b0p1 remains strong, the effect is more consistent with beta sensitivity than pure randomness.
• If alpha endpoints rescue pure TOK-CC at b0p01, the original failure is likely hyperparameter sensitivity rather than a hard beta limitation.
• If dist0p05_tokcc0p95 is strong across alpha endpoints and 1.5B seed replicates, the small distribution-normalized component should remain a candidate for paper figures and compact follow-ups.
• If both pure TOK-CC and small-dist variants vary substantially across seeds, later analysis should treat single-seed method rankings as unstable.

Known Risks

• The downstream seed replicates reuse the same entangled source seed 314159; they test disentangled training stochasticity, not upstream entangled-source variability.
• The split submission uses different hardware pools. Runtime differences should not be interpreted as objective effects.
• The same-job analogical evaluation increases walltime but keeps each run's steering result colocated with its checkpoint.