A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

Citation

• Authors: Iulian V. Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio
• Year: 2017
• Venue: AAAI 2017
• arXiv: arXiv:1605.06069
• URL: https://arxiv.org/abs/1605.06069

One-Line Takeaway

VHRED adds an utterance-level stochastic latent variable to the hierarchical recurrent encoder-decoder model so dialogue generation can represent higher-level uncertainty such as topic or sentiment instead of relying only on token-level sampling noise.

Technical Core

• Let a dialogue be a sequence of utterances $(w_1, \ldots, w_N)$ with utterance $w_n = (w_{n,1}, \ldots, w_{n,M_n})$.
• For each utterance, VHRED introduces a latent variable $z_n$ with prior $$p_\theta(z_n \mid w_{<n}) = \mathcal{N}(\mu_{\mathrm{prior}}(w_{<n}), \Sigma_{\mathrm{prior}}(w_{<n})).$$
• The decoder then generates the next utterance token-by-token: $$p_\theta(w_n \mid z_n, w_{<n}) = \prod_{m=1}^{M_n} p_\theta(w_{n,m} \mid w_{n,<m}, z_n, w_{<n}).$$
• Training uses an approximate posterior $$q_\psi(z_n \mid w_{\le n}) = \mathcal{N}(\mu_{\mathrm{post}}(w_{\le n}), \Sigma_{\mathrm{post}}(w_{\le n}))$$ and maximizes the utterance-factored variational lower bound $$\sum_{n=1}^N \left[ \mathbb{E}_{q_\psi(z_n \mid w_{\le n})}[\log p_\theta(w_n \mid z_n, w_{<n})] - \mathrm{KL}\!\left(q_\psi(z_n \mid w_{\le n}) \,\|\, p_\theta(z_n \mid w_{<n})\right) \right].$$
• The intended role of $z_n$ is to store higher-level utterance information so the decoder hidden state can focus on local token prediction rather than carrying the full long-range conversational trajectory by itself.

Method

• Start from HRED's three-part hierarchy: an encoder RNN for the current utterance, a context RNN for dialogue history, and a decoder RNN for the response.
• Use the context state to parameterize a Gaussian prior over $z_n$.
• Use both the context state and the current-utterance encoding to parameterize a Gaussian approximate posterior over $z_n$ during training.
• Sample $z_n$ and concatenate it with the context representation before feeding the decoder RNN.
• Use several practical anti-collapse heuristics:
  • linearly anneal the KL weight from $0$ to $1$ over the early training phase;
  • drop decoder words at a fixed 25% rate;
  • initialize the encoder and context RNNs from converged HRED models.

Empirical Evaluation

• The paper evaluates dialogue response generation on the Twitter Dialogue Corpus and the Ubuntu Dialogue Corpus.
• Baselines are LSTM, HRED, and a TF-IDF retrieval model.
• On Twitter human evaluation, VHRED is preferred over HRED and TF-IDF for both short and long contexts, and over LSTM for long contexts, while LSTM remains stronger on short contexts because generic "safe" responses are often judged acceptable there.
• On embedding-based similarity metrics, especially in 3-turn generation, VHRED generally tracks the ground-truth response topic better than HRED and LSTM.
• On response-information metrics, VHRED produces higher-entropy and typically longer responses on Twitter, supporting the paper's claim that the latent variable increases diversity and information content.

Assumptions And Limits

The model is designed to improve dialogue generation quality, not to make the latent identifiable in a strong sense. The latent variable is unsupervised and only loosely interpreted as encoding utterance-level factors such as topic or sentiment. The empirical story is also tightly tied to heuristics like KL annealing, word dropout, and HRED initialization, and the evaluation problem itself is noisy because short generic responses can score well in human preference studies.

Relevance To This Project

• Similarities:
  • It is an early example of adding a higher-level latent to an autoregressive sequence generator specifically to model structured uncertainty beyond token noise.
  • It makes the same practical point that matters in this repo: if the decoder is strong, latent usage must be actively protected during training.
  • It is a useful precursor for prompt-conditioned one-to-many generation, even though the conditioning object here is dialogue history rather than a reasoning problem input.
• Differences:
  • VHRED is trained from dialogue data, not as a post-training factorization of an already-capable pretrained conditional language model.
  • Its latent is an utterance-level conversational variable, not a strategy variable whose semantics should stay stable across many different inputs.
  • The paper studies quality and diversity of generations, not disentanglement or cross-input semantic consistency of the latent.

Comparison Notes

Bowman et al. (2016) is the unconditional sequence-VAE precursor that makes posterior collapse in text impossible to ignore; VHRED adds hierarchical context and a conditional generation setting. Sohn et al. (2015) is the cleaner classical conditional-latent factorization, while VHRED is the more direct engineering ancestor for dialogue-style sequence generation. Zhao et al. (2017) can be read as a closer CVAE dialogue follow-up that adds explicit anti-collapse auxiliaries such as bag-of-words prediction.

Related Notes

• Generating Sentences from a Continuous Space
• Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
• CVAE post-training methodology