exp1_gaussian_mixture_high_density-larger-beta-sweep Design

Motivation

This collection is a controlled reference problem for disentanglement. Because the data come from a labeled Gaussian mixture with known ground-truth modes, the sweep can compare discrete and continuous latent VAEs under increasing KL pressure without the confounds present in the language-model experiments.

Setup

• data: labeled Gaussian mixture with K=3, d=2, N=240
• component standard deviation: 0.7
• component separation: 4.0
• beta grid: {0.10, 0.25, 0.50, 1.00, 2.00, 5.00}
• training budget: 20 epochs, batch size 64, optimizer Adam, learning rate 1e-3, seed 17, device cpu

Model Families

• discrete latent VAE:
  • categorical latent with exact softmax-weighted expectation over categories
  • key metrics: loss, recon, kl, nmi, clustering_accuracy
• continuous latent VAE:
  • Gaussian latent with standard normal prior
  • key metrics: loss, recon, kl, linear_probe_accuracy

Research Questions

1. How does increasing KL pressure trade off against reconstruction quality in each latent family?
2. Does the discrete model lose mode-recovery quality as beta grows?
3. Does the continuous model preserve linearly decodable mode information deeper into the beta sweep?