Sum Token Aggregation Ablation: Design

Back to hub

Motivation

Prior Qwen2.5 multi-task reconstruction runs computed supervised-token reconstruction losses with sequence-mean token NLL aggregation. This collection changes that aggregation to sum while keeping the compact 0.5B from-entangled-v2 multi-task setup.

Research Questions

1. Does token-sum aggregation preserve, improve, or degrade router-side disentanglement compared with the prior sequence-mean rows?
2. Under sum aggregation, which reconstruction family and beta setting gives the strongest matched router score?

Sweep

• Reconstruction families: raw, dist_norm, tokcc, dist_plus_tokcc.
• Beta settings: 0.01, 0.1, 1.0.
• Fixed settings: Qwen2.5 0.5B, multi_task, continuous z64 latent, LoRA adaptation, constant beta schedule, one seed.

Metrics

Primary metrics are matched router score, within-task router probe, and final answer accuracy. Supporting metrics include posterior probe, reconstruction losses, and KL-to-prior loss.