ss-disentangled-direct-pretrained-pv-rmkl-sweep-v1 Design

Motivation

This follow-up tests a narrow hypothesis raised by ss-disentangled-direct-pretrained-v1: perhaps the weak latent-strategy alignment is mainly caused by uncontrolled posterior spread or a badly matched router prior. The sweep therefore isolates posterior-variance and router-marginal-KL regularization while keeping the rest of the direct-pretrained setup fixed.

Design

• backbone: pretrained Qwen2.5-0.5B
• training mode: direct disentangling without prior mixture fine-tuning
• adaptation: lora
• latent setup: continuous latent, dimension 8, last-token pooling
• shared settings: beta=0.1, baseline-normalized reconstruction, router_support_weight=0.0, token_weighted_reconstruction_weight=0.0, inter_latent_divergence_weight=0.0
• sweep axes:
  • posterior_variance_weight in {0.0, 0.1}
  • router_marginal_kl_to_prior_weight in {0.0, 0.1}
• training budget: 1 epoch, max_sequences_per_epoch=200000, seed 314159

Research Questions

1. Does reducing posterior variance improve strategy-aligned latent structure?
2. Does forcing the router marginal closer to the prior improve router probe accuracy?
3. If the diagnostics collapse, do answer accuracy and probe quality move with them or stay decoupled?