Neural Discrete Representation Learning

Citation

Authors: Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu
Year: 2017
Venue: NeurIPS 2017
arXiv: arXiv:1711.00937
URL: https://arxiv.org/abs/1711.00937

One-Line Takeaway

VQ-VAE replaces the continuous latent bottleneck with a learned codebook of discrete embeddings, which gives the decoder a latent it cannot as easily ignore and provides a practical route to high-quality discrete latent-variable generative models.

Technical Core

Encoder output $z_e(x)$ is quantized to the nearest codebook vector: $z_q(x) = e_k, \qquad k = \arg\min_j \|z_e(x) - e_j\|_2^2.$
The decoder models $p(x \mid z_q(x))$ .
Training objective: $\log p(x \mid z_q(x)) + \| \mathrm{sg}[z_e(x)] - e \|_2^2 + \beta \| z_e(x) - \mathrm{sg}[e] \|_2^2,$ where sg is stop-gradient.
The second term updates the codebook embeddings, and the third commitment term keeps encoder outputs close to the chosen code.

Method

Use nearest-neighbor vector quantization as the latent bottleneck.
Use a straight-through estimator so gradients from the decoder flow back through the quantized latents to the encoder.
After learning the encoder/decoder/codebook, fit an autoregressive prior such as PixelCNN or WaveNet over the discrete latent codes.

Empirical Evaluation

The paper studies images, video, and raw audio, including CIFAR, ImageNet-style images, DeepMind Lab frames, and speech.
It reports competitive log-likelihood performance and high-quality samples.
In speech, the learned codes behave like high-level phoneme-like content variables and support unsupervised speaker conversion.
One of the headline messages is that the discrete bottleneck largely avoids the posterior-collapse pathology common in continuous VAEs with strong autoregressive decoders.

Assumptions And Limits

The model changes the latent-variable family substantially: it is not just a training tweak to a standard ELBO VAE. Quantization introduces its own architectural and optimization choices, and semantic interpretability of the discrete codes is still empirical rather than guaranteed.

Relevance To This Project

This is one of the most relevant "alternative latent parameterization" papers for the repo. The project already has a discrete-latent branch, and VQ-VAE is a major reference point for why discrete latents can remain active under strong decoders. It is especially useful when thinking about whether latent strategies should be categorical, codebook-based, or token-like rather than Gaussian.

Comparison Notes

Bowman et al. (2016) and He et al. (2019) live in the continuous-latent autoregressive regime and focus on making the latent survive training. VQ-VAE instead changes the bottleneck so that discrete codes and a learned prior become the basic representation. It is also a natural external reference for Discrete latent VAEs.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:1711.00937; no local cache or downloaded source artifacts were created.

Verified from source: title, authors, the nearest-neighbor quantization rule, the three-part training loss including codebook and commitment terms, the learned autoregressive prior over codes, and the broad empirical claims across images / audio / video were checked against the alphaXiv whole-paper report.