Contributions

An initial paper submission focuses on the problem of learning a disentangled factorization of a language model whose distribution is implicitly an entangled mixture of multiple "modes" of reasoning. Applications to exploration, interpretability, etc. are deferred to later papers or versions. The paper is kept self-contained around this general problem.

Positioning: Option A — "We formulate a new problem (latent-variable factorization of pretrained LMs) and initiate its study." The claim is not that the problem is solved; rather, we identify the challenges unique to this setting, develop methodology that addresses them, and provide scaffolding for future methodology to build on.

Headline contributions (three-bullet version)

1. Problem formulation. We formulate the problem of learning a latent-variable router-generator factorization $(r(z|x), g(y|x,z))$ of a pretrained conditional language model $p(y|x)$, and identify the core challenges that make this setting different from standard latent-variable generative modeling — in particular, that the generator at initialization already captures $p(y|x)$ without needing $z$, inducing a pathological form of posterior collapse unique to the adaptation regime.

2. Methodology. We propose a variational training objective that addresses these challenges by placing targeted example-level and token-level pressure on $z$ to encode information about $y$ beyond what is already predictable from $x$, selecting disentangled factorizations that separate strategy modes.

3. Benchmark and empirical evaluation. We introduce a suite of multi-strategy algorithmic tasks with ground-truth latent strategies and empirically evaluate our methodology, including on from-scratch initialized models as well as pretrained language models, characterizing key requirements, ablation effects, and failure modes.

Deferred / conditional contribution

Explicitly excluded from headline contributions

• Exploration proof-of-concept. Per exploration proof of concept, this lives in the discussion section as evidence for future applications, not as a headline claim.
• Applications to interpretability, safety, steerability. Mentioned in intro as motivation and in discussion as future work; not claimed as delivered.