ELBO surgery: yet another way to carve up the variational evidence lower bound

Citation

• Authors: Matthew D. Hoffman and Matthew J. Johnson
• Year: 2016
• Venue: NeurIPS Workshop note
• arXiv:
• URL: https://www.cs.columbia.edu/~blei/fogm/2022F/readings/HoffmanJohnson2016.pdf

One-Line Takeaway

This note decomposes the standard VAE KL term into index-code mutual information plus marginal KL from the aggregate posterior to the prior, making it much clearer what the ELBO is really encouraging.

Technical Core

• Start from the average ELBO term $$\frac{1}{N}\sum_{n=1}^N \mathrm{KL}(q(z \mid x_n)\,\|\,p(z)).$$
• Define the aggregate / average encoder distribution $$q(z) = \frac{1}{N}\sum_{n=1}^N q(z \mid x_n).$$
• Then the note shows $$\frac{1}{N}\sum_{n=1}^N \mathrm{KL}(q(z \mid x_n)\,\|\,p(z)) = I_q(n; z) + \mathrm{KL}(q(z)\,\|\,p(z)),$$ where $n$ is the data index under a uniform distribution over the dataset.
• This yields the three-term decomposition $$\text{ELBO} = \text{average reconstruction} - I_q(n; z) - \mathrm{KL}(q(z)\,\|\,p(z)).$$

Method

This is primarily an algebraic and interpretive note rather than a new training algorithm. Its contribution is to isolate the parts of the ELBO that correspond to data-specific coding versus aggregate prior matching.

Empirical Evaluation

• The note includes a small binarized-MNIST diagnostic showing that the marginal KL term can be a substantial part of the average KL penalty in nontrivial latent dimensions.
• It also reports that the mutual-information term is often already near its maximum, suggesting that a rigid prior may be the bigger bottleneck than insufficient code specificity.

Assumptions And Limits

This note does not propose a new model or guarantee better learning by itself. Its value is conceptual: it clarifies why the same ELBO regularizer can simultaneously behave like an information bottleneck and like a prior-matching penalty.

Relevance To This Project

This note is almost a direct precursor to the repo's own InfoVAE connections note. The current project repeatedly wants to separate "do not erase strategy information" from "keep sampled latents generation-ready." ELBO surgery is one of the cleanest papers for expressing exactly that split.

Comparison Notes

InfoVAE operationalizes this decomposition into a tunable objective family. beta-VAE can be read as globally increasing the whole average-KL term without separating its components. This note explains why that can help in some regimes and hurt in others.

Related Notes

• InfoVAE: Balancing Learning and Inference in Variational Autoencoders
• InfoVAE connections
• CVAE objective and loss terms