Understanding disentangling in beta-VAE

Citation

Authors: Christopher P. Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner
Year: 2018
Venue: arXiv preprint
arXiv: arXiv:1804.03599
URL: https://arxiv.org/abs/1804.03599

One-Line Takeaway

This paper explains beta-VAE in rate-distortion terms and argues that disentanglement is better controlled by gradually increasing latent capacity than by fixing a single large KL weight forever.

Technical Core

The paper starts from the beta-VAE objective $\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \beta\, \mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)).$
It interprets the KL term as the information-carrying capacity of the latent bottleneck.
The paper's preferred variant instead uses a target capacity $C$ and penalty weight $\gamma$ : $\mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - \gamma\, \big|\mathrm{KL}(q_\phi(z \mid x)\,\|\,p(z)) - C\big|.$
Training starts with small $C$ , so the model must allocate scarce capacity to the most salient factors first, then gradually increases $C$ so additional factors can enter without destroying previously learned separation.

Method

Recast disentanglement as a capacity-allocation problem.
Increase the latent information budget over training rather than imposing a fixed high-pressure bottleneck.
Use latent traversals and known generative factors to inspect how different factors are encoded as capacity grows.

Empirical Evaluation

The paper reports results on dSprites, colored dSprites, and 3D Chairs.
A main finding is that different factors are encoded in a staged order as capacity increases.
Compared with a fixed-beta regime, controlled-capacity schedules preserve stronger reconstruction quality while still obtaining clean factor separation.

Assumptions And Limits

Like beta-VAE itself, this remains a heuristic disentanglement method rather than an identification theorem. It explains and improves a useful empirical phenomenon, but it still assumes that the data and architecture provide a regime where bottlenecked factors separate in a useful way.

Relevance To This Project

This paper is relevant because it provides a more nuanced language than "turn beta up or down." Many of the repo's current loss questions are really about latent capacity allocation: how much information should $z$ carry, when should that information become available, and how can one keep structure without destroying reconstruction or generation quality?

Comparison Notes

Higgins et al. (2017) introduced the simpler beta-weighted view. This paper explains why that works and why it can degrade reconstruction. InfoVAE instead removes or reshapes the mutual-information penalty when the problem is that $z$ carries too little information. The two papers are complementary because one studies controlled bottlenecks and the other studies decoupled bottlenecks.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:1804.03599; no local cache or downloaded source artifacts were created.

Verified from source: title, authors, the beta-VAE objective, the capacity-controlled objective with target $C$ , the staged-capacity interpretation, and the dSprites / colored dSprites / 3D Chairs experimental role were checked against the alphaXiv whole-paper report.