ss-disentangled-direct-pretrained-v1 Design

Motivation

This collection is a small direct-pretrained diagnostic meant to answer a narrow question before broader sweeps: if disentangling starts directly from the pretrained Qwen2.5-0.5B backbone, does the latent already become strategy-aligned under the baseline reconstruction-plus-KL objective?

Design

• backbone: pretrained Qwen2.5-0.5B
• training mode: direct disentangling without prior mixture fine-tuning
• adaptation: lora
• latent setup: continuous latent, dimension 8, last-token pooling
• training budget: 1 epoch, max_sequences_per_epoch=200000, seed 314159
• optimizer: adamw, lr=2e-4
• active objective: baseline-normalized reconstruction plus beta=0.1 KL-to-prior
• diagnostic-only terms: posterior_variance_weight=0.0, router_marginal_kl_to_prior_weight=0.0, router_support_weight=0.0, token_weighted_reconstruction_weight=0.0, inter_latent_divergence_weight=0.0

Research Questions

1. Does the baseline objective yield a clean strategy-coded latent without additional regularization?
2. If not, is the weakness visible on both router-sampled and posterior-side probes?
3. Do the logged but inactive auxiliary diagnostics already show a mismatch between objective optimization and latent-structure quality?

Scope And Risks

• planned design: 6 tasks x 1 setting x 1 seed = 6 jobs
• observed completion at analysis time: 5/6 complete, with sorting_algorithms excluded after CUDA OOM
• key metrics: sampled final-answer accuracy, router probe accuracy, posterior probe accuracy, normalized reconstruction, KL prior, posterior variance, router marginal KL