LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces

Citation

• Authors: Yingji Zhang, Danilo S. Carvalho, Ian Pratt-Hartmann, and Andre Freitas
• Year: 2023
• Venue: arXiv preprint
• arXiv: arXiv:2312.13208
• URL: https://arxiv.org/abs/2312.13208
• Citation count: 1 (NLP Knowledge Graph, retrieved 2026-04-20)

One-Line Takeaway

LlaMaVAE modernizes the Optimus line by replacing the encoder and decoder with sentenceT5 and LLaMA, then uses the resulting latent sentence space for smoother semantic control and conditional generation.

Technical Core

• The encoder maps a sentence $x$ into a Gaussian latent posterior $$q_\phi(z \mid x) = \mathcal{N}(\mu(x), \Sigma(x)),$$ using a pretrained sentenceT5 representation.
• A sampled latent vector $z$ is expanded by an MLP and injected into each LLaMA decoder layer as an additional memory-like conditioning signal.
• Training follows a standard VAE objective $$\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \beta \, \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)),$$ together with collapse-mitigation heuristics inherited from prior language-VAE work.
• The paper also introduces an Invertible CVAE variant, where an invertible neural network learns a mapping between external conditioning representations and the VAE latent sentence space.

Method

• Replace the older BERT/GPT-2 stack used by Optimus with stronger pretrained encoder/decoder components.
• Freeze most LLaMA hidden layers during pretraining and let the latent-conditioning path supply sentence-level control.
• Evaluate the latent space both as an encoder representation space and as a generation-control interface.
• For definition modeling, use an invertible neural network to map between lexical embeddings and the pretrained latent sentence space, enabling conditional generation in both directions.

Empirical Evaluation

• LlaMaVAE outperforms Optimus on the paper's language-modeling and reconstruction-style evaluations across the studied corpora.
• On semantic textual similarity and probing tasks, it generally improves over Optimus, though not every latent-space use case dominates specialized embedding methods.
• Latent interpolation and traversal are qualitatively smoother and more semantically consistent than in earlier language VAEs.
• In the definition-modeling setting, the invertible-CVAE variant improves over Optimus, though the paper still reports a noticeable generalization gap when moving beyond the training data regime.

Assumptions And Limits

The latent variable here is still a sentence-level semantic control variable, not an explicitly disentangled reasoning variable. The paper improves geometric smoothness and controllability, but it does not establish that the latent coordinates correspond to stable causal reasoning modes. The invertible-conditioning mechanism also inherits the usual risk that geometric alignment looks cleaner than actual compositional generalization.

Relevance To This Project

This is one of the closest nearby papers to the repo's broad motivation. It explicitly asks how modern LLM generation can be routed through a continuous latent sentence space instead of being controlled only through prompts or token prefixes.

Relative to the repo, though, the objective is still different:

• LlaMaVAE wants semantically controllable generation from a sentence latent;
• this repo wants a factorization of an existing conditional model into latent strategy routing and latent-conditioned generation;
• the repo's latent variable is supposed to track abstract strategy rather than generic sentence semantics.

Comparison Notes

Optimus is the direct architectural ancestor. Park and Lee (2021) focuses on low-resource finetuning into a VAE, whereas LlaMaVAE returns to a stronger-pretraining style story with better modern components. Zhang et al. (2025) pushes this line closer to explicit reasoning-rule supervision rather than generic latent sentence control.

Related Notes

• OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space
• Finetuning Pretrained Transformers into Variational Autoencoders
• Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study
• Abstract Strategy vs Solution