Learning Structured Output Representation using Deep Conditional Generative Models

Citation

• Authors: Kihyuk Sohn, Honglak Lee, and Xinchen Yan
• Year: 2015
• Venue: NeurIPS 2015
• arXiv:
• URL: https://papers.nips.cc/paper/5775-learning-structured-output-representation-using-deep-conditional-generative-models

One-Line Takeaway

This paper is the clean early CVAE reference: it models multi-modal structured outputs by combining a conditional prior, a latent-conditioned generator, and a recognition model trained with a conditional ELBO.

Technical Core

• Conditional latent-variable model: $$p_\theta(y, z \mid x) = p_\theta(z \mid x)\, p_\theta(y \mid x, z).$$
• Variational approximation: $$q_\phi(z \mid x, y).$$
• Conditional ELBO: $$\log p_\theta(y \mid x) \ge \mathbb{E}_{q_\phi(z \mid x, y)}[\log p_\theta(y \mid x, z)] - \mathrm{KL}(q_\phi(z \mid x, y)\,\|\,p_\theta(z \mid x)).$$
• The key modeling point is that $x$ conditions both the prior over latent modes and the decoder, so the latent can capture ambiguity in the output $y$ rather than ambiguity in the input representation itself.

Method

• Use three networks with distinct roles:
  • a prior network for $p_\theta(z \mid x)$,
  • a recognition network for $q_\phi(z \mid x, y)$,
  • and a generation network for $p_\theta(y \mid x, z)$.
• Train with SGVB-style reparameterized gradients.
• Add practical robustness tricks for structured prediction, including input noise injection and multi-scale prediction objectives.
• The paper also studies a Gaussian stochastic neural network variant and a hybrid objective combining CVAE and GSNN-style training.

Empirical Evaluation

• Experiments focus on structured prediction tasks such as semantic labeling and image segmentation on LFW and CUB-style benchmarks.
• The main empirical claim is that the latent conditional model can produce diverse but realistic outputs where deterministic feed-forward predictors are forced into one averaged answer.
• The architecture and training additions are presented as complementary to the conditional latent-variable formulation.

Assumptions And Limits

The paper is written for ambiguous structured outputs in vision. It does not address the later sequence-model collapse story in autoregressive decoders, and it does not try to make the latent identifiable in a strong theoretical sense.

Relevance To This Project

This is the closest classical ancestor of the repo's training factorization.

• Similarities:
  • The paper uses the same conditional latent-variable decomposition that the repo now studies: $$p_\phi(z \mid x), \qquad p_\psi(y \mid x, z), \qquad q_\xi(z \mid x, y).$$
  • The latent is meant to explain residual variation in $y$ after conditioning on $x$.
  • Training proceeds through a conditional ELBO with a learned prior and a learned recognition model.
• Differences:
  • The paper's target is generic one-to-many structured prediction, not disentangling latent reasoning strategies inside an already-capable generator.
  • The model is trained from scratch on supervised data, whereas the repo's methodology often starts from a strong entangled generator and asks how to refactor or post-train it.
  • There is no explicit pressure here for latent semantics to align with reusable strategy structure beyond representing output ambiguity.

Comparison Notes

Kingma and Welling (2013) is the unconditional precursor. Bowman et al. (2016) moves the same latent-variable idea into autoregressive sequence modeling, where collapse becomes much more severe. He et al. (2019) can be read as addressing one major failure mode that this formulation later exhibits in strong decoders.

Related Notes

• Auto-Encoding Variational Bayes
• CVAE problem setup
• CVAE model components