Generating Sentences from a Continuous Space

Citation

• Authors: Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio
• Year: 2016
• Venue: CoNLL 2016
• arXiv: arXiv:1511.06349
• URL: https://aclanthology.org/K16-1002/

One-Line Takeaway

This is the classic sentence-VAE paper: it shows how to add a global continuous latent variable to an autoregressive language model, and it also makes the posterior-collapse problem concrete in sequence modeling.

Technical Core

• Sentence model: $$p(z) = \mathcal{N}(0, I), \qquad p_\theta(x \mid z) = \prod_t p_\theta(x_t \mid x_{<t}, z).$$
• Inference network: $$q_\phi(z \mid x).$$
• ELBO: $$\mathcal{L}(x) = \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)).$$
• The decoder is an RNN language model conditioned on a global latent code, so the latent is meant to capture sentence-level properties such as topic or style.

Method

• Encode the input sentence into a variational posterior over a continuous latent variable.
• Decode with an RNN conditioned on the sampled latent.
• Use KL cost annealing so the model does not immediately collapse to ignoring $z$.
• Use word dropout on decoder inputs so the autoregressive path cannot rely entirely on previous ground-truth tokens.

Empirical Evaluation

• The paper studies unconditional sentence generation, interpolation in latent space, missing-word imputation, and downstream classification-style uses of the learned representation.
• It reports that latent traversals and sentence interpolations are qualitatively meaningful.
• It also reports a negative result that the sentence VAE is not necessarily a better language model than a strong plain RNNLM, which is itself part of the collapse story.

Assumptions And Limits

The paper establishes the appeal of global sequence latents, but it also reveals how hard they are to train when paired with strong autoregressive decoders. The mitigation tricks are heuristic and somewhat sensitive to schedule and architecture.

Relevance To This Project

This is a key bridge paper between classical VAEs and the repo's actual sequence setting.

• Similarities:
  • Both settings want a global latent variable to modulate behavior in an autoregressive sequence generator.
  • Both face the same core optimization hazard: a strong decoder can explain most of the sequence without using $z$.
  • Both therefore treat latent usage, not just output quality, as a central problem.
• Differences:
  • Bowman et al. study an unconditional sentence model $p(x)$ rather than a conditional factorization of $p(y \mid x)$.
  • Their latent is intended to capture broad sentence properties such as topic or style, not explicit reasoning strategy modes tied to a prompt.
  • The repo's work is closer to a conditional post-training or refactorization problem, where the baseline generator already models $p(y \mid x)$ before latent training begins.

Comparison Notes

Sohn et al. (2015) is the cleaner CVAE ancestor for conditional outputs, but this paper is more relevant for sequence decoders. He et al. (2019) directly continues the collapse story opened here. VQ-VAE is a very different answer to the same problem: use a discrete bottleneck that strong decoders are less able to ignore.

Related Notes

• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Synthetic sequences experiments
• Discrete latent VAEs