Auto-Encoding Variational Bayes

Citation

• Authors: Diederik P. Kingma and Max Welling
• Year: 2013
• Venue: arXiv preprint; later ICLR 2014
• arXiv: arXiv:1312.6114
• URL: https://arxiv.org/abs/1312.6114

One-Line Takeaway

This paper introduced the modern VAE recipe: optimize an ELBO for a deep latent-variable model while using a reparameterized inference network so both the decoder and encoder can be trained by backpropagation.

Technical Core

• Generative model: $$p_\theta(x, z) = p(z)\, p_\theta(x \mid z).$$
• Variational approximation: $$q_\phi(z \mid x).$$
• Evidence lower bound: $$\log p_\theta(x) \ge \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)).$$
• Main computational idea: write a latent sample as $$z = g_\phi(\epsilon, x), \qquad \epsilon \sim p(\epsilon),$$ so gradients move through the stochastic node by differentiating through the deterministic map $g_\phi$.

Method

• Learn a probabilistic encoder $q_\phi(z \mid x)$ and decoder $p_\theta(x \mid z)$ jointly.
• Use the stochastic gradient variational Bayes estimator to optimize the ELBO on minibatches.
• In the standard Gaussian case, set $$z = \mu_\phi(x) + \sigma_\phi(x) \odot \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I),$$ which makes the latent draw differentiable with respect to encoder parameters.
• When the prior and approximate posterior are chosen as Gaussians, the KL term is analytic, leaving only the reconstruction expectation to estimate.

Empirical Evaluation

• The paper evaluates the method on MNIST and Frey Face.
• Relative to Wake-Sleep style training, the proposed estimator reaches better variational lower bounds and produces smooth latent interpolations.
• The experiments establish the practical feasibility of jointly learning a deep generative model and an amortized inference network with ordinary gradient methods.

Assumptions And Limits

The core estimator is cleanest for continuous reparameterizable latents. The paper does not solve identifiability, disentanglement, or posterior-collapse issues; it provides the basic optimization framework on top of which those later questions are posed.

Relevance To This Project

This is the base reference for the repo's ELBO-based CVAE setup. The current factorization

$$p_\phi(z \mid x)\, p_\psi(y \mid x, z), \qquad q_\xi(z \mid x, y),$$

is a conditional extension of the same basic variational picture recorded in CVAE problem setup and CVAE objective and loss terms.

Comparison Notes

Sohn, Lee, and Yan (2015) lifts this unconditional VAE recipe to the conditional structured-output setting closest to the repo's methodology. Zhao, Song, and Ermon (2019) and Hoffman and Johnson (2016) analyze what the KL term is really doing beyond this original formulation.

Related Notes

• Learning Structured Output Representation using Deep Conditional Generative Models
• CVAE problem setup
• CVAE objective and loss terms