beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Citation

• Authors: Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner
• Year: 2017
• Venue: ICLR 2017
• arXiv:
• URL: https://openreview.net/forum?id=Sy2fzU9gl

One-Line Takeaway

beta-VAE made the simple but influential claim that increasing the weight on the KL term can force a VAE to use a lower-capacity, more factorized latent representation, often yielding better unsupervised disentanglement.

Technical Core

• Starting from the standard VAE ELBO $$\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)),$$ the paper introduces the modified objective $$\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \beta\, \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)),$$ with $\beta > 1$.
• The intended effect is to tighten the latent information bottleneck so individual latent dimensions are pushed toward a more independent, compressed coding of the data-generating factors.

Method

• Keep the VAE architecture essentially standard.
• Change only the relative weight on the KL term through a scalar $\beta$.
• Evaluate disentanglement through latent traversals and a weakly supervised factor-prediction protocol.

Empirical Evaluation

• The paper reports qualitative and quantitative gains over a standard VAE, InfoGAN, and DC-IGN on visual datasets such as faces, chairs, and CelebA-style settings.
• The main empirical message is that suitably tuned $\beta > 1$ often improves factor separation, though at some cost to reconstruction fidelity.

Assumptions And Limits

The method is heuristic rather than identifiable in a theorem-level sense. It trades reconstruction against latent structure, and it does not distinguish the mutual-information and aggregate-matching roles of the KL term. Later papers make both of those limitations much more explicit.

Relevance To This Project

This is the canonical reference for the intuition that stronger KL pressure can encourage more structured latents. In the current repo, that same intuition appears whenever the loss discussion asks whether increasing or decreasing the effective KL pressure helps latent strategies become cleaner. It is the simplest baseline answer to "make $z$ more bottlenecked and hopefully more interpretable."

Comparison Notes

Burgess et al. (2018) explains this paper in a rate-distortion language and proposes a better capacity-controlled variant. InfoVAE takes almost the opposite direction in settings where the KL term suppresses useful latent information. Khemakhem et al. (2020) is the sharp contrast on identifiability: beta-VAE encourages factorization heuristically, while iVAE adds assumptions under which the latent structure becomes identifiable.

Related Notes

• Understanding disentangling in beta-VAE
• CVAE objective and loss terms
• Identifiability context