Lagging Inference Networks and Posterior Collapse in Variational Autoencoders

Citation

• Authors: Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick
• Year: 2019
• Venue: ICLR 2019
• arXiv: arXiv:1901.05534
• URL: https://arxiv.org/abs/1901.05534

One-Line Takeaway

This paper argues that posterior collapse is not only an objective-design problem but also a training-dynamics problem: the amortized inference network lags behind the true posterior early in training, which pushes the model toward ignoring the latent.

Technical Core

• Standard VAE marginal: $$p_\theta(x) = \int p_\theta(x \mid z)\, p(z)\, dz.$$
• Standard training optimizes the ELBO with amortized posterior $q_\phi(z \mid x)$.
• The paper's diagnosis is geometric: early in training, $q_\phi(z \mid x)$ poorly tracks the moving target $p_\theta(z \mid x)$, so updates to the generator are made against a weak posterior approximation and drift toward a collapsed solution with near-zero mutual information between $x$ and $z$.

Method

• Keep the VAE model and ELBO objective unchanged.
• Modify optimization by taking aggressive extra encoder updates before each model update whenever the current latent mutual information is too low.
• The method is intended to reduce "inference lag" without adding new architectural machinery.

Empirical Evaluation

• The paper studies synthetic geometry, text datasets such as Yahoo and Yelp, and image settings such as Omniglot.
• Relative to basic VAE training, the aggressive-inference schedule yields higher KL and mutual-information usage, more active latent dimensions, and better held-out likelihood in strong autoregressive settings.
• It is also positioned as much cheaper than semi-amortized alternatives that add local inference loops.

Assumptions And Limits

The method does not change what the optimum means; it changes which basin training reaches. It helps when collapse is caused by optimization trajectory, but it does not by itself solve aggregate-prior mismatch or identifiability.

Relevance To This Project

This is highly relevant because the repo's target setting also uses a strong conditional decoder that can already solve much of the prediction problem without consulting $z$.

• Similarities:
  • Both settings live in the regime where the decoder is strong enough to make latent underuse or collapse a first-order risk.
  • The paper's diagnosis that optimization trajectory matters, not just the static objective, matches the repo's concern that entangled initialization can trap training near latent-ignoring solutions.
  • The sequence setting in the paper is especially relevant for language-model-like generators.
• Differences:
  • He et al. study standard VAE training for $p(x)$, not a conditional router-generator factorization of $p(y \mid x)$.
  • Their proposed fix changes optimization dynamics while keeping the ELBO objective unchanged; the repo also studies objective modifications that explicitly prefer strategy-bearing factorizations.
  • The paper is about keeping the latent active at all, not about aligning it with identifiable or interpretable strategy structure.

Comparison Notes

Bowman et al. (2016) is the classic sequence-VAE paper that first made the collapse problem vivid in autoregressive language models. InfoVAE attacks collapse by changing the objective's information geometry. This paper instead argues that even the standard ELBO can work much better if the encoder is trained aggressively enough to keep up with the generator.

Related Notes

• Generating Sentences from a Continuous Space
• Posterior variance
• CVAE objective and loss terms