Exp1 Gaussian Mixture: Larger Beta Sweep (Concise Report)

Exp1 Gaussian Mixture: Larger Beta Sweep (Concise Report)

Artifact source: path:experiments/gaussian_mixture/demos/artifacts/exp1_gaussian_mixture_high_density-larger-beta-sweep

Methodology

Notation

• $x \in \mathbb{R}^d$ is an observed sample and $z \in \{1, \ldots, K\}$ is a latent mode.
• $\mathbb{E}$ denotes expectation, $\mathrm{KL}$ denotes KL divergence, and $\mathrm{MSE}$ denotes mean squared error.
• $\mathcal{N}$ denotes a Gaussian distribution and $\mathrm{Unif}$ denotes a uniform categorical prior.

Synthetic data is sampled from a labeled Gaussian mixture: $z \sim \mathrm{Cat}(\pi), \quad x \mid z = k \sim \mathcal{N}(\mu_k, \sigma^2 I_d), \quad x \in \mathbb{R}^d.$ The larger sweep uses $K=3$, $d=2$, $N=240$, and $\beta \in \{0.10, 0.25, 0.50, 1.00, 2.00, 5.00\}.$

Model definitions (priors + objectives)

Both variants optimize $L(x) = L_\mathrm{recon}(x) + \beta L_\mathrm{kl} (x).$

• Discrete latent VAE

  • Posterior: $q_\phi(z = k \mid x) = \operatorname{softmax}(f_\phi(x))_k, \quad k \in \{1, \ldots, K\},$
  • Prior: uniform categorical $p(z) = \mathrm{Unif}([K]).$
  • Implementation note: training uses exact softmax-weighted expectation over categories (no Gumbel-softmax sampling).
  • Objective: $L_\mathrm{disc}(x) = \sum_{k=1}^{K} q_\phi(z = k \mid x)\, \mathrm{MSE}(x, \hat{x}_k) + \beta \mathrm{KL}(q_\phi(z \mid x) \,\|\, p(z)).$
  • Reported history/summary metrics: loss, recon, kl, nmi, clustering_accuracy.

• Continuous latent VAE

  • Posterior: $q_\phi(z \mid x) = \mathcal{N}(\mu_\phi(x), \operatorname{diag}(\sigma_\phi(x)^2)).$
  • Prior: standard Gaussian $p(z) = \mathcal{N}(0, I).$
  • Objective: $L_\mathrm{cont}(x) = \mathbb{E}_{q_\phi(z \mid x)}[\mathrm{MSE}(x, \hat{x}(z))] + \beta \mathrm{KL}(q_\phi(z \mid x) \,\|\, p(z)).$
  • Reported history/summary metrics: loss, recon, kl, linear_probe_accuracy.

Metric definitions

• recon: mean squared reconstruction error.
• kl: KL divergence term in the objective.
• nmi: normalized mutual information between predicted and true modes (discrete).
• clustering_accuracy: permutation-invariant label alignment accuracy $\max_{\pi \in S_K} \frac{1}{N} \sum_{i} \mathbf{1}[\pi(\hat{z}_i) = z_i].$
• linear_probe_accuracy: holdout linear classifier accuracy from continuous latent means to true modes.

Experimental Setup

Quantitative results

Discrete latent summary

Continuous latent summary

Figures

Caption: Discrete training history metrics across beta.

Caption: Continuous training history metrics across beta.

Caption: Discrete metric heatmap and confusion matrices.

Caption: Reconstruction vs KL tradeoff across latent type and beta.

Key takeaways

• Discrete mode recovery (NMI/accuracy) is strongest at lower beta, while KL pressure increases reconstruction cost as beta grows.
• Continuous latents remain highly linearly predictive of true modes across the sweep (probe accuracy ~1.0 except slight drop at beta=5.0).
• Larger beta compresses continuous latent geometry (mean radius drops from 2.935 to 0.271), consistent with stronger prior matching.