ELBO objectives and max-info factorizations

Setup

Let $X,Y,Z$ be finite random variables. Let $\mu$ be a fixed input law on $\mathcal X$, and let

$$p_0(y\mid x)$$

be the data conditional law. Write

$$P_0(x,y):=\mu(x)p_0(y\mid x).$$

Let

$$H_0:=H_{P_0}(Y\mid X)=\mathbb E_{P_0}[-\log p_0(Y\mid X)]$$

be the base conditional entropy. Unless stated otherwise, logarithms are natural.

We consider an encoder, prior/router, and decoder

$$q(z\mid x,y),\qquad r(z\mid x),\qquad g(y\mid x,z).$$

The encoder induces the joint law

$$Q_q(x,y,z):=\mu(x)p_0(y\mid x)q(z\mid x,y).$$

All quantities with subscript $q$ are computed under $Q_q$.

Define the reconstruction loss and prior KL loss

$$\operatorname{Rec}(q,g) :=\mathbb E_{Q_q}[-\log g(Y\mid X,Z)],$$ $$\operatorname{KL}(q\Vert r) :=\mathbb E_{X,Y\sim P_0}\operatorname{KL}\!\left(q(\cdot\mid X,Y)\Vert r(\cdot\mid X)\right).$$

The standard negative $\beta$-ELBO objective is

$$\mathcal L_\beta(q,r,g) := \operatorname{Rec}(q,g)+\beta\,\operatorname{KL}(q\Vert r).$$

The earlier max-info theory studies exact factorizations

$$p_0(y\mid x)=\sum_z r(z\mid x)g(y\mid x,z)$$

and the criterion $\max I(Y;Z\mid X)$. The central identity there is

$$I(Y;Z\mid X)=H(Y\mid X)-H(Y\mid X,Z),$$

so max-info is equivalent to minimizing residual generator entropy.

Encoder-induced factorization

Given $q(z\mid x,y)$, define its marginal and reverse conditional

$$q(z\mid x):=\sum_y p_0(y\mid x)q(z\mid x,y),$$ $$q(y\mid x,z):= \frac{p_0(y\mid x)q(z\mid x,y)}{q(z\mid x)}$$

whenever $q(z\mid x)>0$.

\begin{proof} For the first claim,

$$\sum_z q(z\mid x)q(y\mid x,z) = \sum_z q(z\mid x) \frac{p_0(y\mid x)q(z\mid x,y)}{q(z\mid x)} = p_0(y\mid x)\sum_z q(z\mid x,y) = p_0(y\mid x).$$

For the converse, direct substitution gives

$$\sum_y p_0(y\mid x)q_{r,g}(z\mid x,y) = \sum_y r(z\mid x)g(y\mid x,z) = r(z\mid x),$$

and therefore

$$q_{r,g}(y\mid x,z) = \frac{p_0(y\mid x)q_{r,g}(z\mid x,y)}{r(z\mid x)} = g(y\mid x,z).$$

The mutual informations are equal because the induced joint laws on $(X,Y,Z)$ are identical. \end{proof}

Thus variational encoders and exact latent factorizations are dual descriptions of the same object when the prior and decoder are chosen to match the encoder-induced conditionals.

Reconstruction and KL decompositions

Define the decoder mismatch and prior mismatch

$$\operatorname{DecGap}(q,g) := \mathbb E_{X,Z\sim Q_q} \operatorname{KL}\!\left(q(\cdot\mid X,Z)\Vert g(\cdot\mid X,Z)\right),$$ $$\operatorname{PriorGap}(q,r) := \mathbb E_{X\sim\mu} \operatorname{KL}\!\left(q(\cdot\mid X)\Vert r(\cdot\mid X)\right).$$

\begin{proof} First,

$$\begin{aligned} \operatorname{Rec}(q,g) &= \mathbb E_{X,Z}\mathbb E_{Y\sim q(\cdot\mid X,Z)}[-\log g(Y\mid X,Z)]\\ &= \mathbb E_{X,Z} \left[ H(q(\cdot\mid X,Z)) + \operatorname{KL}\!\left(q(\cdot\mid X,Z)\Vert g(\cdot\mid X,Z)\right) \right]\\ &= H_q(Y\mid X,Z)+\operatorname{DecGap}(q,g). \end{aligned}$$

Because the $(X,Y)$-marginal of $Q_q$ is $P_0$,

$$H_q(Y\mid X)=H_0.$$

Thus

$$H_q(Y\mid X,Z)=H_0-I_q(Y;Z\mid X),$$

which proves the reconstruction identity.

For the KL term,

$$\begin{aligned} \operatorname{KL}(q\Vert r) &= \mathbb E_{X,Y,Z\sim Q_q} \log \frac{q(Z\mid X,Y)}{r(Z\mid X)}\\ &= \mathbb E_{Q_q} \log \frac{q(Z\mid X,Y)}{q(Z\mid X)} + \mathbb E_{Q_q} \log \frac{q(Z\mid X)}{r(Z\mid X)}\\ &= I_q(Y;Z\mid X) + \mathbb E_X\operatorname{KL}\!\left(q(\cdot\mid X)\Vert r(\cdot\mid X)\right). \end{aligned}$$

Combining the two identities gives the stated formula for $\mathcal L_\beta$. \end{proof}

\begin{proof} The mismatch terms are nonnegative and are simultaneously zero at $r=r_q$ and $g=g_q$. Apply the theorem. \end{proof}

Regimes of the standard $\beta$-ELBO

Assume $r$ and $g$ are expressive enough to match every encoder-induced factorization in the encoder class $\mathcal Q$. Let

$$I_{\max}:=\sup_{q\in\mathcal Q} I_q(Y;Z\mid X), \qquad I_{\min}:=\inf_{q\in\mathcal Q} I_q(Y;Z\mid X).$$

\begin{proof} By the previous corollary,

$$\inf_{r,g}\mathcal L_\beta(q,r,g) = H_0+(\beta-1)I_q(Y;Z\mid X).$$

For \beta<1, the coefficient of $I_q(Y;Z\mid X)$ is negative, so minimization selects maximal information. For $\beta>1$, the coefficient is positive, so minimization selects minimal information. For $\beta=1$, the coefficient is zero, so the optimized value is $H_0$ for every $q$. \end{proof}

Relation to exact max-info factorizations

Suppose $(r,g)$ is an exact factorization of $p_0$:

$$p_0(y\mid x)=\sum_z r(z\mid x)g(y\mid x,z).$$

Let $q_{r,g}$ be its exact posterior.

\begin{proof} Under the posterior encoder $q_{r,g}$, the induced joint is

$$\mu(x)r(z\mid x)g(y\mid x,z).$$

Thus $Y\mid X,Z$ has law $g(\cdot\mid X,Z)$, giving

$$\operatorname{Rec}(q_{r,g},g) = \mathbb E[-\log g(Y\mid X,Z)] = H(Y\mid X,Z).$$

Also, because $q_{r,g}(z\mid x)=r(z\mid x)$, the prior mismatch term is zero, so the KL decomposition gives

$$\operatorname{KL}(q_{r,g}\Vert r)=I(Y;Z\mid X).$$

Finally,

$$H(Y\mid X,Z)=H_0-I(Y;Z\mid X).$$

\end{proof}

Therefore, on exact factorizations:

\beta<1 \quad\Longleftrightarrow\quad \text{max-info selection}, $$\beta=1 \quad\Longleftrightarrow\quad \text{all exact factorizations tie},$$ $$\beta>1 \quad\Longleftrightarrow\quad \text{min-info selection}.$$

Likelihood-bound interpretation at $\beta=1$

For fixed $r,g$, define the model marginal

$$m_{r,g}(y\mid x):=\sum_z r(z\mid x)g(y\mid x,z)$$

and its posterior

$$\pi_{r,g}(z\mid x,y) := \frac{r(z\mid x)g(y\mid x,z)}{m_{r,g}(y\mid x)}.$$

\begin{proof} Pointwise in $(x,y)$,

$$\begin{aligned} &\mathbb E_q[-\log g(y\mid x,Z)] + \operatorname{KL}\!\left(q(\cdot\mid x,y)\Vert r(\cdot\mid x)\right)\\ &=\sum_z q(z\mid x,y) \left[-\log g(y\mid x,z)+\log\frac{q(z\mid x,y)}{r(z\mid x)}\right]\\ &=\sum_z q(z\mid x,y) \log\frac{q(z\mid x,y)}{r(z\mid x)g(y\mid x,z)}\\ &=\sum_z q(z\mid x,y) \log\frac{q(z\mid x,y)}{\pi_{r,g}(z\mid x,y)m_{r,g}(y\mid x)}\\ &= \operatorname{KL}\!\left(q(\cdot\mid x,y)\Vert \pi_{r,g}(\cdot\mid x,y)\right) - \log m_{r,g}(y\mid x). \end{aligned}$$

Averaging over $(X,Y)\sim P_0$ gives the result. \end{proof}

Thus $\beta=1$ is the likelihood-fitting regime. It does not prefer a more informative or less informative exact latent factorization unless additional restrictions break the tie.

Capacity-constrained regime

Suppose $|\mathcal Z|=m$. Then

$$I_q(Y;Z\mid X)\le H_q(Z\mid X)\le \log m.$$

Therefore, after optimizing $r,g$,

$$\operatorname{Rec}^\star(q)=H_0-I_q(Y;Z\mid X) \ge H_0-\min\{H_0,\log m\}.$$

If the encoder is a hard task-conditioned attribute

$$Z=\phi(X,Y),$$

then

$$I_q(Y;Z\mid X)=H_q(Z\mid X).$$

Therefore

$$\inf_{r,g}\mathcal L_\beta(q,r,g) = H_0+(\beta-1)H(\phi(X,Y)\mid X).$$

This is the ELBO analogue of the capacity-constrained max-info theory: in the low-$\beta$ regime, the objective discovers the highest-entropy admissible branches of the output distribution. To make those branches semantic, one still needs structural assumptions such as a hierarchical branch class, cross-input alignment, or a generator simplicity penalty.

Normalized reconstruction and the role of $H_0$

A normalized reconstruction loss is

$$\operatorname{NRec}(q,g) := \frac{\operatorname{Rec}(q,g)}{H_0}.$$

At the optimized decoder $g=q(y\mid x,z)$,

$$\operatorname{NRec}^\star(q) = \frac{H_0-I_q(Y;Z\mid X)}{H_0} = 1-\frac{I_q(Y;Z\mid X)}{H_0}.$$

Thus normalized reconstruction has a meaningful information-theoretic interpretation:

• $\operatorname{NRec}=1$ means $I_q(Y;Z\mid X)=0$;
• $\operatorname{NRec}=0$ means $H_q(Y\mid X,Z)=0$;
• $\operatorname{NRec}\in(0,1)$ is the fraction of conditional uncertainty in $Y$ that remains after observing $Z$.

With decoder misspecification,

$$\operatorname{NRec}(q,g) = 1-\frac{I_q(Y;Z\mid X)}{H_0} +\frac{\operatorname{DecGap}(q,g)}{H_0},$$

so it is an upper bound on the residual uncertainty fraction.

Therefore $H_0$ is meaningful as a scale for interpreting reconstruction as a fractional residual uncertainty. But if only the reconstruction term is divided by $H_0$, the numerical value and units of $\beta$ change. For cross-problem comparison, normalizing the entire objective or explicitly setting $\beta$ relative to $1/H_0$ is cleaner.

Conditional versus unconditional prior

The above analysis assumes a conditional prior/router $r(z\mid x)$. If instead the prior is unconditional, $r(z)$, then

$$\mathbb E_{X,Y}\operatorname{KL}\!\left(q(\cdot\mid X,Y)\Vert r(\cdot)\right)$$

penalizes more than $I_q(Y;Z\mid X)$.

\begin{proof} Decompose the log ratio:

$$\log\frac{q(Z\mid X,Y)}{r(Z)} = \log\frac{q(Z\mid X,Y)}{q(Z\mid X)} + \log\frac{q(Z\mid X)}{q(Z)} + \log\frac{q(Z)}{r(Z)}.$$

Taking expectations gives the three terms. \end{proof}

Thus an unconditional prior also penalizes dependence between $X$ and $Z$. For strategy modeling, this may be undesirable if different tasks genuinely have different strategy mixtures. A conditional prior $r(z\mid x)$ isolates the information that $Z$ carries about $Y$ beyond $X$.

Empirical objective and sample complexity

Let

$$h:=(q,r,g)$$

and define the per-sample loss

$$\ell_h(x,y) := \sum_z q(z\mid x,y)[-\log g(y\mid x,z)] + \beta\sum_z q(z\mid x,y) \log\frac{q(z\mid x,y)}{r(z\mid x)}.$$

The population and empirical objectives are

$$\mathcal L_\beta(h)=\mathbb E_{P_0}\ell_h(X,Y),$$ $$\widehat{\mathcal L}_\beta(h)=\frac1n\sum_{i=1}^n \ell_h(x_i,y_i), \qquad (x_i,y_i)\overset{iid}{\sim}P_0.$$

Assume a bounded-loss condition: for all $h$ in the candidate class $\mathcal H$,

$$0\le \ell_h(x,y)\le B.$$

A sufficient support condition is

$$g(y\mid x,z)\ge \gamma>0, \qquad r(z\mid x)\ge \alpha>0$$

whenever the corresponding terms are evaluated. Then one may take

$$B=\log\frac1\gamma+\beta\log\frac1\alpha.$$

\begin{proof} For fixed $h$, Hoeffding's inequality gives

$$\mathbb P\left( \left| \widehat{\mathcal L}_\beta(h)-\mathcal L_\beta(h) \right|>\varepsilon \right) \le 2\exp\left(-\frac{2n\varepsilon^2}{B^2}\right).$$

A union bound over $N$ hypotheses gives the first claim after setting the right-hand side equal to $\delta$.

On the event of uniform deviation at most $\varepsilon$,

$$\mathcal L_\beta(\widehat h) \le \widehat{\mathcal L}_\beta(\widehat h)+\varepsilon \le \widehat{\mathcal L}_\beta(h^\star)+\varepsilon \le \mathcal L_\beta(h^\star)+2\varepsilon.$$

Substituting the value of $\varepsilon$ proves the excess-risk bound. \end{proof}

\begin{proof} The previous theorem gives uniform deviation at most

$$\varepsilon=B\sqrt{\frac{\log(2N/\delta)}{2n}}.$$

The displayed sample-size condition implies 2\varepsilon<\Delta. For $h\notin\mathcal E^\star$ and $h^\star\in\mathcal E^\star$,

$$\widehat{\mathcal L}_\beta(h) \ge \mathcal L_\beta(h)-\varepsilon \ge \mathcal L_\beta(h^\star)+\Delta-\varepsilon > \mathcal L_\beta(h^\star)+\varepsilon \ge \widehat{\mathcal L}_\beta(h^\star).$$

Thus no empirical minimizer can lie outside $\mathcal E^\star$. \end{proof}

For infinite model classes, the same structure holds with the finite-class term replaced by a complexity measure for the induced loss class

$$\mathfrak L:=\{\ell_h:h\in\mathcal H\}.$$

A typical Rademacher-complexity bound has the form

$$\sup_{h\in\mathcal H} \left| \widehat{\mathcal L}_\beta(h)-\mathcal L_\beta(h) \right| \lesssim \mathfrak R_n(\mathfrak L) +B\sqrt{\frac{\log(1/\delta)}{n}}.$$

Thus optimizing the empirical ELBO to accuracy $\varepsilon$ requires roughly

$$n=\widetilde O\!\left(\frac{\operatorname{complexity}(\mathfrak L)}{\varepsilon^2}\right),$$

and recovering a separated optimum requires roughly

$$n=\widetilde O\!\left(\frac{\operatorname{complexity}(\mathfrak L)}{\Delta^2}\right).$$

Monte Carlo estimation over $z$

If $\mathcal Z$ is small, the expectation over $q(z\mid x,y)$ can be computed exactly. If it is estimated using $M$ independent samples of $Z$ per datapoint, the reconstruction estimate adds Monte Carlo noise.

Assume the reconstruction term is bounded by $B_{\mathrm{rec}}$. Then, conditionally on the data, averaging over $nM$ independent latent samples gives an additional error of order

$$O\!\left(B_{\mathrm{rec}}\sqrt{\frac{\log(1/\delta)}{nM}}\right)$$

by Hoeffding's inequality. This adds to the statistical error from sampling $(X,Y)$.

Consequences for semantic strategy recovery

The ELBO analysis connects directly to the capacity-constrained semantic-strategy theory.

1. In the low-$\beta$ regime, the optimized ELBO is a max-info objective.
2. With hard encoders and finite $|\mathcal Z|$, this becomes a search for the highest-entropy admissible branch attribute.
3. Under hierarchical branch assumptions, this can recover local branch structure up to $x$-dependent permutation.
4. The ELBO objective alone does not solve global semantic alignment; one still needs a cross-input alignment, equivariance, or simplicity penalty.
5. In the standard $\beta=1$ likelihood regime, exact factorizations tie, so likelihood alone does not identify the semantic latent.
6. In the $\beta>1$ regime, the objective actively discourages conditional information in $Z$ and tends toward posterior collapse unless constrained otherwise.

Summary

The standard negative $\beta$-ELBO decomposes as

$$\mathcal L_\beta = H_0+(\beta-1)I_q(Y;Z\mid X) +\text{decoder mismatch} +\beta\cdot\text{prior mismatch}.$$

Consequently:

• \beta<1: information-seeking / max-info regime;
• $\beta=1$: likelihood-fitting regime; all exact factorizations tie;
• $\beta>1$: information-bottleneck / min-info regime.

Normalized reconstruction by $H_0$ is useful for interpreting reconstruction as a fraction of remaining conditional uncertainty, but it changes the numerical threshold for $\beta$ unless the KL term is scaled consistently. Statistical recovery of the population-selected factorization requires bounded losses, a controlled hypothesis class, and a positive gap between the desired factorization class and its competitors.

References

1. D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes, 2013.
2. A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy, Deep Variational Information Bottleneck, 2016.
3. The companion note Max-info factorizations of a conditional distribution for the exact max-info / Jensen–Shannon identity and deterministic-factorization degeneracy.