Latent Thought Models with Variational Bayes Inference-Time Computation

Citation

• Authors: Deqian Kong, Minglu Zhao, Dehong Xu, Bo Pang, Shu Wang, Edouardo Honig, Zhangzhang Si, Chuan Li, Jianwen Xie, Sirui Xie, and Ying Nian Wu
• Year: 2025
• Venue: ICML 2025
• arXiv: arXiv:2502.01567
• URL: https://proceedings.mlr.press/v267/kong25c.html
• Citation count: 1 (ResearchGate preprint record for the same DOI, retrieved 2026-04-20)

One-Line Takeaway

Latent Thought Models introduce explicit continuous thought vectors and optimize them at inference time, arguing that inference steps and latent-thought count create new scaling axes beyond model size alone.

Technical Core

• Introduce latent thought vectors $z$ with a simple prior $$p(z) = \mathcal{N}(0, I),$$ and an autoregressive decoder $$p_\beta(x \mid z).$$
• For each training sequence $x$, use a sequence-specific variational posterior $$q(z \mid x) = \mathcal{N}(\mu, \sigma^2),$$ and optimize the local posterior parameters directly by gradient descent.
• This yields a classical variational-Bayes ELBO with dual-rate optimization:
  • fast updates for local latent posterior parameters;
  • slow updates for shared decoder parameters.
• The latent thought vectors are injected layerwise into a Transformer decoder through cross-attention, so generation depends on both recent token context and a global latent thought state.

Method

• Train a decoder with a relatively short context window so the latent thought vectors must carry global information.
• Treat the number of fast inference steps $T_{\mathrm{fast}}$ and the number of latent thought vectors $N_z$ as explicit scaling dimensions.
• Use inference-time optimization of $q(z \mid x)$ both during training and for conditional generation.
• Compare against autoregressive and discrete-diffusion baselines under matched compute-oriented views.

Empirical Evaluation

• LTMs improve validation perplexity and zero-shot language-modeling performance relative to matched autoregressive and discrete-diffusion baselines.
• The paper reports strong sample and parameter efficiency: more latent-thought inference can partially substitute for simply scaling parameter count.
• It also reports emergent few-shot arithmetic reasoning at smaller model scales than the authors argue would typically be needed with standard autoregressive models.
• Conditional and unconditional generation quality remains competitive while sampling can be faster than some baseline alternatives.

Assumptions And Limits

This is not a disentanglement paper in the repo's sense. The latent state is useful for generation and efficiency, but it is not claimed to recover human-readable reasoning modes or reusable abstract strategies. The model also pays a real price in inference-time optimization complexity: the latent state is not just sampled once, it is iteratively optimized.

Relevance To This Project

The paper is highly relevant as a nearby alternative answer to the question "what should a latent variable do in a language model?" Here, the answer is: it should act like an optimized hidden thought state that improves generation and compute efficiency.

That is close to the repo's interests, but not identical:

• LTMs focus on performance, scaling, and inference-time computation;
• the repo focuses on factorizing an existing model into latent strategies that can be interpreted, routed, and intervened on;
• LTMs use continuous latent thought states without claiming they correspond to disentangled reasoning rules.

Comparison Notes

Compared with Optimus and LlaMaVAE, this paper is less about sentence-level semantic control and more about latent-variable computation as an alternative language-modeling architecture. Compared with Zhang et al. (2025), it does not supervise explicit reasoning-rule structure in the latent space.

Related Notes

• OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space
• Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study
• Abstract Strategy vs Solution
• Challenges of Learning a Latent Variable Factorization of Pre-trained Language Models