ELBO Under-Specification And The Z-Ignoring Solution

This note records the formal failure-mode argument for the standard negative conditional ELBO in the adapted-from-base router-generator setting. The key point is that the standard variational objective is non-identifying: it can preserve the observable conditional law $p_\theta(y|x)$ while leaving the latent variable $z$ inert.

Reference phrasing:

The standard negative conditional ELBO is non-identifying in the adapted-from-base setting. If the generator can represent the base model while ignoring $z$, then $g_\phi(y|x,z)=p_\theta(y|x)$ and $q_\xi(z|x,y)=r_\phi(z|x)$ attain the global optimum of the $\beta=1$ objective. Thus preserving the observable law $p_\theta(y|x)$ does not require $z$ to encode strategy information. Since our generator is initialized near this z-ignoring solution, posterior collapse is not merely a generic VAE pathology; it is the natural absorbing solution of the pretrained factorization problem.

Core Distinction

There are two related but distinct problems:

• Objective-level under-specification: distributional fidelity to $p_\theta(y|x)$ does not identify a strategy-aligned router-generator factorization. Many factorizations can induce the same latent marginal $p_\phi(y|x)$.
• Adapted-from-base optimization pathology: because the generator is initialized from the base model, the z-ignoring factorization is not only possible; it is nearby at initialization. Standard variational training can therefore remain near a collapsed solution.

Use z-ignoring factorization for the concrete failure mode where $g_\phi(y|x,z)$ does not depend on $z$.

Use absorbing z-ignoring solution or optimization sink for the adapted-from-base training pathology. Prefer absorbing solution in the main paper unless a strict local-stability argument is given.

Population Objective

Let the training distribution be

$$x\sim \mathcal D_X, \qquad y\sim p_\theta(\cdot|x).$$

The standard negative conditional ELBO is

$$\mathcal J_{\mathrm{cELBO}}(x,y;\phi,\xi) = \mathbb E_{z\sim q_\xi(\cdot|x,y)} \left[ -\log g_\phi(y|x,z) \right] + \mathrm{KL} \left( q_\xi(\cdot|x,y) \Vert r_\phi(\cdot|x) \right).$$

The population objective is

$$\mathcal L_{\mathrm{cELBO}}(\phi,\xi) = \mathbb E_{x\sim\mathcal D_X,\;y\sim p_\theta(\cdot|x)} \left[ \mathcal J_{\mathrm{cELBO}}(x,y;\phi,\xi) \right].$$

For a router-generator pair $(r_\phi,g_\phi)$, define the induced latent marginal

$$p_\phi(y|x) = \int r_\phi(z|x)g_\phi(y|x,z)\,d\mu(z).$$

When $p_\phi(y|x)>0$, define the exact posterior induced by the factorization:

$$p_\phi(z|x,y) = \frac{ r_\phi(z|x)g_\phi(y|x,z) }{ p_\phi(y|x) }.$$

ELBO Identity

A simple ELBO identity makes the under-specification explicit. For fixed $(x,y)$, the $\beta=1$ negative conditional ELBO can be rewritten as

$$\mathcal J_{\mathrm{cELBO}}(x,y;\phi,\xi) = -\log p_\phi(y|x) + \mathrm{KL} \left( q_\xi(\cdot|x,y) \Vert p_\phi(\cdot|x,y) \right).$$

Thus the standard objective first asks the induced marginal $p_\phi(y|x)$ to match the target conditional law and then asks $q_\xi$ to match the posterior of whichever factorization produced that marginal. It does not by itself select for a posterior or latent variable with strategy semantics.

Call this an observation, calculation, or ELBO identity in paper prose rather than a proposition unless the section needs formal theorem-like structure.

Degenerate Optimum Observation

If the generator class can represent the base conditional law while ignoring $z$, then the z-ignoring factorization is a global minimizer of the $\beta=1$ population negative conditional ELBO.

More explicitly, for any $\phi,\xi$,

$$\mathcal L_{\mathrm{cELBO}}(\phi,\xi) \ge H_\theta(Y|X),$$

where

$$H_\theta(Y|X) = \mathbb E_{x\sim \mathcal D_X,\;y\sim p_\theta(\cdot|x)} \left[ -\log p_\theta(y|x) \right].$$

Equality is attained whenever $p_\phi(y|x)=p_\theta(y|x)$ and $q_\xi(\cdot|x,y)=p_\phi(\cdot|x,y)$ almost surely. In particular, equality is attained by the z-ignoring factorization

$$g_\phi(y|x,z)=p_\theta(y|x) \quad\text{for all }z, \qquad q_\xi(z|x,y)=r_\phi(z|x).$$

Derivation

For fixed $(x,y)$, use the induced posterior $p_\phi(z|x,y)$ to rewrite the per-example objective:

$$\begin{aligned} &\mathbb E_{z\sim q_\xi(\cdot|x,y)} \left[ -\log g_\phi(y|x,z) \right] + \mathrm{KL} \left( q_\xi(\cdot|x,y) \Vert r_\phi(\cdot|x) \right) \\ &= \mathbb E_{z\sim q_\xi(\cdot|x,y)} \left[ \log \frac{ q_\xi(z|x,y) }{ r_\phi(z|x)g_\phi(y|x,z) } \right] \\ &= -\log p_\phi(y|x) + \mathrm{KL} \left( q_\xi(\cdot|x,y) \Vert p_\phi(\cdot|x,y) \right). \end{aligned}$$

Taking expectation over $x\sim\mathcal D_X$ and $y\sim p_\theta(\cdot|x)$ gives

$$\mathcal L_{\mathrm{cELBO}}(\phi,\xi) = \mathbb E_{x,y\sim p_\theta} \left[ -\log p_\phi(y|x) \right] + \mathbb E_{x,y\sim p_\theta} \left[ \mathrm{KL} \left( q_\xi(\cdot|x,y) \Vert p_\phi(\cdot|x,y) \right) \right].$$

The first term is the cross-entropy from $p_\theta(\cdot|x)$ to $p_\phi(\cdot|x)$:

$$\mathbb E_{x,y\sim p_\theta} \left[ -\log p_\phi(y|x) \right] = H_\theta(Y|X) + \mathbb E_{x\sim\mathcal D_X} \left[ \mathrm{KL} \left( p_\theta(\cdot|x) \Vert p_\phi(\cdot|x) \right) \right].$$

Both KL terms are nonnegative, so

$$\mathcal L_{\mathrm{cELBO}}(\phi,\xi) \ge H_\theta(Y|X).$$

Equality holds when $p_\phi(y|x)=p_\theta(y|x)$ and $q_\xi(\cdot|x,y)=p_\phi(\cdot|x,y)$ almost surely.

For the z-ignoring factorization $g_\phi(y|x,z)=p_\theta(y|x)$, the induced marginal is $p_\phi(y|x)=p_\theta(y|x)$ and the induced posterior is $p_\phi(z|x,y)=r_\phi(z|x)$. Therefore choosing $q_\xi(z|x,y)=r_\phi(z|x)$ attains equality. This proves that the z-ignoring factorization is a global minimizer.

Beta-Weighted Variant

For an exact factorization $p_\phi(y|x)=p_\theta(y|x)$ with exact posterior $q_\xi(z|x,y)=p_\phi(z|x,y)$, the beta-weighted negative conditional ELBO satisfies

$$\mathcal L_\beta = H_\theta(Y|X) + (\beta-1)I_\phi(Y;Z|X).$$

Thus:

• $\beta=1$: all exact factorizations tie, including the z-ignoring factorization;
• $\beta>1$: lower-information factorizations are favored, so collapse is selected among exact factorizations;
• \beta<1: higher-information factorizations are favored, but this relies on weakening KL pressure and does not by itself solve strategy alignment or adapted-from-base optimization.

This beta-weighted identity is useful context, but the main paper can use the simpler $\beta=1$ observation as the core under-specification argument.

If using softer language, describe the $\beta=1$ result as the ELBO under-specification observation and the $\beta$-weighted result as a related information identity.

Optimization Consequence

At the z-ignoring solution:

• the generator already matches the target conditional law $p_\theta(y|x)$;
• the reconstruction term is independent of $z$, so it gives no signal that distinguishes latent values;
• the KL term is minimized by $q_\xi(z|x,y)=r_\phi(z|x)$;
• therefore the posterior receives no reconstruction incentive to encode information about $y$ beyond $x$.

This is why posterior collapse is unusually severe in the adapted-from-base setting. In ordinary VAE training from random initialization, the generator initially fails to predict $y$, so reconstruction can create pressure for the latent to carry useful information. Here, the generator begins near a solution that predicts $y$ from $x$ alone, so the collapsed factorization is immediately available.

Reference phrasing:

Standard variational training can preserve the observable conditional distribution while leaving the latent variable inert. The issue is not that the negative conditional ELBO cannot fit $p_\theta(y|x)$; it can fit it through the z-ignoring factorization. The issue is that this objective does not select among marginally equivalent factorizations for one in which $z$ carries strategy information.

Link To Base-Directed Reconstruction

This observation motivates Base-directed reconstruction objective. The proposed $p_\theta$-directed reconstruction term $\mathcal R_\theta$ changes the reconstruction pressure so that training is measured in z-free base-model reference units and concentrated on tokens where the base model has residual surprisal.

Reference transition:

The ELBO identity shows that the standard variational objective is under-specified for the factorization problem: matching the base marginal does not force $z$ to be informative. We therefore replace the uniform reconstruction term with a $p_\theta$-directed reconstruction pressure that asks whether conditioning on $z$ explains residual surprise beyond what $x$ already predicts under the base model.

Softer version:

This identity shows that the standard variational objective answers the wrong selection question for our setting. It asks whether the router-generator pair can reproduce the observable conditional law, but not whether the latent variable accounts for any of the variation in $y$ that remains after conditioning on $x$. Because the base model already predicts $y$ from $x$, a uniform reconstruction term can be satisfied near initialization without using $z$. We therefore replace it with a $p_\theta$-directed reconstruction pressure that measures reconstruction in z-free base-model reference units and concentrates token-level pressure where the base model has residual surprisal.

Paper Message Flow

A compact presentation of this concept can use this logic:

1. Introduce the standard negative conditional ELBO with explicit router-generator notation.
2. State the $\beta=1$ ELBO identity: $$\mathcal J_{\mathrm{cELBO}}(x,y;\phi,\xi) = -\log p_\phi(y|x) + \mathrm{KL}(q_\xi(\cdot|x,y)\Vert p_\phi(\cdot|x,y)).$$
3. Explain the under-specification: in expectation over $y\sim p_\theta(\cdot|x)$, the objective is minimized by matching the induced marginal $p_\phi(y|x)$ to $p_\theta(y|x)$ and matching $q_\xi$ to the induced posterior, but this does not require a strategy-aligned latent.
4. Exhibit the z-ignoring optimum $g_\phi(y|x,z)=p_\theta(y|x)$ and $q_\xi(z|x,y)=r_\phi(z|x)$.
5. Explain why the adapted-from-base setting makes this collapse unusually accessible: the generator begins near the z-ignoring solution, so reconstruction is already low and KL can collapse the posterior toward the router.
6. Transition to Base-directed reconstruction objective: uniform reconstruction predicts $y$ from $(x,z)$, while $\mathcal R_\theta$ asks for reconstruction pressure in units and positions defined by the z-free base model.

Terms That Need Care

Posterior collapse is the familiar VAE term and should be used for the observed optimization outcome, but it does not by itself explain the adapted-from-base pathology.

Latent collapse is a reasonable synonym, but posterior collapse is more standard.

Naive ELBO is casual. Prefer standard negative conditional ELBO or standard variational objective in the main paper.

Sink is vivid, but it can imply a strict dynamical-systems stability result. Prefer absorbing z-ignoring solution, collapse basin, or optimization sink unless local stability is proved.

Under-specified objective and non-identifying objective are both useful. Use under-specified for reader-facing prose and non-identifying when making the formal identification point.