InfoVAE: Balancing Learning and Inference in Variational Autoencoders

Citation

• Authors: Shengjia Zhao, Jiaming Song, and Stefano Ermon
• Year: 2019
• Venue: AAAI 2019
• arXiv: arXiv:1706.02262
• URL: https://arxiv.org/abs/1706.02262

One-Line Takeaway

InfoVAE separates two jobs that the standard VAE KL term mixes together: discouraging information flow through $z$ and matching the aggregate posterior to the prior.

Technical Core

• Let $$q_\phi(z) = \int q_\phi(z \mid x)\, p_D(x)\, dx$$ denote the aggregate posterior.
• The paper's key claim is that the usual ELBO KL term conflates a mutual-information penalty with aggregate posterior matching.
• One equivalent InfoVAE family can be written as $$\mathcal{L}_{\mathrm{InfoVAE}} = \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - (1-\alpha)\,\mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)) - (\alpha + \lambda - 1)\, D(q_\phi(z), p(z)),$$ where $D$ can be a divergence such as MMD or an adversarially estimated discrepancy.
• The parameter $\alpha$ controls how strongly the model penalizes latent information, while the aggregate-matching term keeps samples from the prior generation-ready.

Method

• Replace the standard ELBO's one-term regularizer with two tunable pieces:
  • a per-example KL term, and
  • a separate discrepancy between the aggregate posterior and the prior.
• Instantiate $D$ in several ways, including MMD and adversarial matching.
• Use these controls to avoid the "ignore $z$" failure mode when the decoder is too flexible.

Empirical Evaluation

• The paper evaluates variants such as MMD-VAE on binarized MNIST and CIFAR-style settings.
• Reported gains include better latent usage, improved aggregate-prior matching, and stronger semi-supervised features relative to a standard ELBO baseline.
• MMD-based variants are presented as especially stable compared with adversarial matching.

Assumptions And Limits

InfoVAE is still a training-objective intervention, not an identifiability theorem. It improves the odds that the latent carries useful information, but it does not by itself force that information to align with semantically meaningful factors or strategies.

Relevance To This Project

This is one of the most directly relevant papers in the whole batch because the repo already reframes its current loss stack in InfoVAE language in InfoVAE connections. The mapping is almost explicit: low beta reduces the mutual-information penalty, while the router-marginal regularizer plays the role of aggregate prior matching.

Comparison Notes

Hoffman and Johnson (2016) provides the algebraic decomposition that makes the mutual-information versus marginal-matching split easy to see. beta-VAE emphasizes stronger KL pressure, whereas InfoVAE emphasizes that in many regimes the KL is already penalizing exactly the information one wants to keep. He et al. (2019) studies a complementary collapse story based on training dynamics rather than objective decomposition.

Related Notes

• ELBO surgery
• InfoVAE connections
• Router marginal KL