Literature: Variational Autoencoders
Literature: Variational Autoencoders
Papers on variational autoencoders and their variants relevant to the project. Scope includes:
- foundational VAEs (Kingma and Welling, 2013);
- identifiability and nonlinear ICA framings of VAEs (Khemakhem et al., 2020);
- disentanglement-oriented variants (beta-VAE and follow-ups);
- posterior collapse diagnosis and mitigation;
- VAEs applied to autoregressive or sequence models;
- conditional VAEs and other structured-latent extensions.
- adjacent latent-variable disentanglement baselines that are not themselves VAEs but are useful comparison points for loss design.
Papers
Foundations And Conditional Formulations
- Kingma and Welling (2013) - Auto-Encoding Variational Bayes
- Sohn, Lee, and Yan (2015) - Learning Structured Output Representation using Deep Conditional Generative Models
- Bowman et al. (2016) - Generating Sentences from a Continuous Space
These are the best entry points for the basic modeling story: the original VAE, the classical CVAE factorization, and the first influential sequence-VAE paper where autoregressive collapse becomes impossible to ignore.
Conditional Outputs And One-To-Many Generation
- Sohn, Lee, and Yan (2015) - Learning Structured Output Representation using Deep Conditional Generative Models
- Serban et al. (2017) - A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues
- Kohl et al. (2018) - A Probabilistic U-Net for Segmentation of Ambiguous Images
- Zhao, Zhao, and Eskenazi (2017) - Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
This cluster is the closest background for the repo's conditional factorization story. Sohn gives the basic CVAE decomposition, Serban et al. show how a hierarchical utterance-level latent can improve dialogue generation, Kohl shows how a conditional latent can represent coherent ambiguous outputs rather than averaged predictions, and Zhao et al. move the same idea into conditional text generation with more explicit anti-collapse heuristics.
ELBO Geometry, Mutual Information, And Disentanglement
- Chen et al. (2016) - InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
- Higgins et al. (2017) - beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework
- Kim and Mnih (2018) - Disentangling by Factorising
- Hoffman and Johnson (2016) - ELBO surgery: yet another way to carve up the variational evidence lower bound
- Burgess et al. (2018) - Understanding disentangling in beta-VAE
- Zhao, Song, and Ermon (2019) - InfoVAE: Balancing Learning and Inference in Variational Autoencoders
This cluster is the most directly relevant for the repo's current loss design. It separates six distinct ideas that are easy to blur together: explicit mutual-information rewards for latent recoverability, stronger KL bottlenecks, explicit total-correlation penalties, capacity schedules, algebraic decomposition of the ELBO, and aggregate-posterior matching without erasing mutual information.
InfoGAN explicitly rewards latent recoverability, beta-VAE strengthens the whole KL bottleneck, FactorVAE isolates factorization pressure through total correlation, ELBO surgery decomposes the regularizer into interpretable pieces, Burgess et al. turn bottleneck strength into a capacity schedule, and InfoVAE separates mutual-information pressure from aggregate matching.
Collapse, Discrete Latents, And Identifiability
- He et al. (2019) - Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- Fu et al. (2019) - Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
- van den Oord, Vinyals, and Kavukcuoglu (2017) - Neural Discrete Representation Learning
- Khemakhem et al. (2020) - Variational Autoencoders and Nonlinear ICA: A Unifying Framework
These papers are useful when the question is not just "how do I train a latent model?" but "why does the latent stay active?" and "when does the latent correspond to something real rather than an arbitrary reparameterization?" He et al. emphasize optimization dynamics, Fu et al. emphasize KL scheduling in autoregressive text models, VQ-VAE changes the bottleneck type, and iVAE sharpens the identifiability story.
Pretrained Language VAEs And Latent Reasoning Models
- Li et al. (2020) - OPTIMUS: Organizing Sentences via Pre-trained Modeling of a Latent Space
- Park and Lee (2021) - Finetuning Pretrained Transformers into Variational Autoencoders
- Zhang et al. (2023) - LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces
- Kong et al. (2025) - Latent Thought Models with Variational Bayes Inference-Time Computation
- Zhang et al. (2025) - Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study
This cluster tracks the path from classical sentence VAEs to modern latent-variable language models that are explicitly discussed as control, reasoning, or latent-thought systems. Optimus scales the basic recipe with large pretrained components, Park and Lee show what can be done with finetuning-only adaptation, LlaMaVAE modernizes the architecture for stronger LLMs, Latent Thought Models treat the latent as an optimized thought state, and Zhang et al. push closest to explicitly supervised latent reasoning-rule separation.
