• 1. Introduction
• 2. Setup and recovery metrics
• 3. Expressive ELBO and profiled router-generator cost
• 4. Profiled recovery margin
• 5. Marginal fidelity and the small-complexity limit
• 6. Deterministic hard-strategy specialization
• 7. Empirical recovery
• 8. Interpretation and limits

Unified variational complexity strategy recovery

1. Introduction

This note studies when a variational latent-variable objective can recover a reference strategy structure, not merely fit the observable conditional law $p(y\mid x)$. The updated setup treats the realistic case where the ELBO is optimized over independently parameterized encoders, routers, and generators:

$$q(z\mid x,y)\in\mathcal Q,\qquad r(z\mid x)\in\mathcal R,\qquad g(y\mid x,z)\in\mathcal G.$$

The complexity term is a joint router-generator penalty $C(r,g)$ rather than a decoder-only penalty. This better matches the factorization-selection story: a strategy should make both the routing rule and the strategy-conditioned generator simple.

The guiding principle becomes:

$$\boxed{ \text{low-}\beta\text{ ELBO supplies information pressure, while profiled router-generator complexity supplies selection.} }$$

The deterministic completed-trace attribute theorem is still useful, but it is now a specialization. The main theorem is over stochastic encoders $q\in \mathcal Q$ and uses the effective profiled cost

$$A_{\beta,\lambda}(q) := \inf_{r\in\mathcal R,g\in\mathcal G} \left[ \operatorname{DecGap}(q,g) + \beta\operatorname{PriorGap}(q,r) + \lambda C(r,g) \right].$$

This is an approximation-complexity regularizer: it penalizes encoders whose induced routing law and branch conditionals are hard to approximate faithfully with a simple router-generator pair.

2. Setup and recovery metrics

Let $\mathcal X$ and $\mathcal Y$ be finite. Let $\mu$ be a fixed input law and let $p(y\mid x)$ be the fixed conditional law being factorized. All expectations, entropies, and mutual informations are computed under

$$X\sim\mu,\qquad Y\sim p(\cdot\mid X)$$

unless stated otherwise. Fix $\mathcal Z=[K]$.

For an encoder $q(z\mid x,y)$, define

$$Q_q(x,y,z)=\mu(x)p(y\mid x)q(z\mid x,y).$$

The encoder-induced router and branch conditional are

$$q(z\mid x)=\sum_y p(y\mid x)q(z\mid x,y),$$

and, when $q(z\mid x)>0$,

$$q(y\mid x,z)= \frac{p(y\mid x)q(z\mid x,y)}{q(z\mid x)}.$$

Let $I_q(Y;Z\mid X)$ denote conditional mutual information under $Q_q$.

Let the reference strategy be $S^\star=\sigma(X,Y)\in[K]$. The hard-strategy assumption says that the strategy is recognizable from the completed trace $(X,Y)$. Define the hard reference encoder

$$q^\star(z\mid x,y)=\mathbf 1\{z=\sigma(x,y)\},$$

and assume $q^\star\in\mathcal Q$ when stating recovery theorems.

For a stochastic encoder, define weak and strong recovery distances under $Q_q$:

$$d_{\mathrm w}(q,S^\star) := H_q(Z\mid X,S^\star)+H_q(S^\star\mid X,Z),$$

and

$$d_{\mathrm s}(q,S^\star) := H_q(Z\mid S^\star)+H_q(S^\star\mid Z).$$

Weak recovery allows an input-dependent relabeling of latent states. Strong recovery requires one global relabeling.

\begin{proof} The proof is the same variation-of-information argument as in the deterministic case. For weak recovery, $d_{\mathrm w}=0$ if and only if

$$H_q(Z\mid X,S^\star)=0, \qquad H_q(S^\star\mid X,Z)=0.$$

Thus, for $\mu$-almost every $x$, $Z$ is a deterministic function of $S^\star$ and $S^\star$ is a deterministic function of $Z$ under $Q_q(\cdot\mid X=x)$. Hence the positive-probability labels match up to a permutation $\pi_x$. The converse is immediate. The strong statement is the same argument without conditioning on $X$. Finally, conditioning reduces entropy, giving $d_{\mathrm w}\le d_{\mathrm s}$. \end{proof}

3. Expressive ELBO and profiled router-generator cost

The joint-complexity beta-ELBO is

$$\mathcal L_{\beta,\lambda}(q,r,g) := \mathbb E_{Q_q}[-\log g(Y\mid X,Z)] + \beta\,\mathbb E_{X,Y} \operatorname{KL}(q(\cdot\mid X,Y)\|r(\cdot\mid X)) + \lambda C(r,g).$$

Define

$$\operatorname{DecGap}(q,g) := \mathbb E_{X,Z} \operatorname{KL}(q(\cdot\mid X,Z)\|g(\cdot\mid X,Z)),$$

where the first argument is the encoder-induced conditional law over $Y$, and

$$\operatorname{PriorGap}(q,r) := \mathbb E_X \operatorname{KL}(q(\cdot\mid X)\|r(\cdot\mid X)).$$

\begin{proof} The reconstruction term decomposes as cross-entropy:

$$\mathbb E[-\log g(Y\mid X,Z)] = H_q(Y\mid X,Z)+\operatorname{DecGap}(q,g).$$

Since the $(X,Y)$ marginal of $Q_q$ is $\mu(x)p(y\mid x)$,

$$H_q(Y\mid X,Z) = H(Y\mid X)-I_q(Y;Z\mid X).$$

The KL-to-prior term decomposes by inserting $q(Z\mid X)$:

$$\mathbb E_{X,Y}\operatorname{KL}(q(\cdot\mid X,Y)\|r(\cdot\mid X)) = I_q(Y;Z\mid X)+\operatorname{PriorGap}(q,r).$$

Combining the terms proves the identity. \end{proof}

For a fixed encoder, define the profiled router-generator cost

$$A_{\beta,\lambda}(q) := \inf_{r\in\mathcal R,g\in\mathcal G} \left[ \operatorname{DecGap}(q,g) + \beta\operatorname{PriorGap}(q,r) + \lambda C(r,g) \right],$$

and the profiled loss

$$\bar{\mathcal L}_{\beta,\lambda}(q) := \inf_{r,g} \mathcal L_{\beta,\lambda}(q,r,g).$$

Then

$$\bar{\mathcal L}_{\beta,\lambda}(q) = H(Y\mid X) - (1-\beta)I_q(Y;Z\mid X) + A_{\beta,\lambda}(q).$$

For \beta<1, define

$$J^{\mathrm{prof}}_{\beta,\lambda}(q) := I_q(Y;Z\mid X) - \frac{A_{\beta,\lambda}(q)}{1-\beta}.$$

Then

$$\bar{\mathcal L}_{\beta,\lambda}(q) = H(Y\mid X) - (1-\beta)J^{\mathrm{prof}}_{\beta,\lambda}(q).$$

So full ELBO optimization over $(q,r,g)$ is equivalent, after profiling out the router and generator, to maximizing the profiled information-complexity score over encoders.

4. Profiled recovery margin

For $\rho\in\{\mathrm w,\mathrm s\}$ and $\delta>0$, define

$$\mathcal B_\rho(\delta) := \{q\in\mathcal Q:d_\rho(q,S^\star)\ge\delta\}.$$

The profiled recovery margin is

$$\Delta_{\beta,\lambda}^{\rho,\mathrm{prof}}(\delta) := J^{\mathrm{prof}}_{\beta,\lambda}(q^\star) - \sup_{q\in\mathcal B_\rho(\delta)} J^{\mathrm{prof}}_{\beta,\lambda}(q).$$

\begin{proof} Since \beta<1,

$$\bar{\mathcal L}_{\beta,\lambda}(q) = H(Y\mid X) - (1-\beta)J^{\mathrm{prof}}_{\beta,\lambda}(q),$$

so minimizing the profiled loss is equivalent to maximizing $J^{\mathrm{prof}}_{\beta,\lambda}$. Any profiled minimizer satisfies

$$J^{\mathrm{prof}}_{\beta,\lambda}(\widehat q) \ge J^{\mathrm{prof}}_{\beta,\lambda}(q^\star).$$

If $d_\rho(\widehat q,S^\star)\ge\delta$, then $\widehat q\in\mathcal B_\rho(\delta)$ and therefore

$$J^{\mathrm{prof}}_{\beta,\lambda}(\widehat q) \le J^{\mathrm{prof}}_{\beta,\lambda}(q^\star) - \Delta_{\beta,\lambda}^{\rho,\mathrm{prof}}(\delta),$$

a contradiction. A full ELBO minimizer induces a profiled minimizer in $q$, because any lower profiled value at its encoder, or at another encoder, would give a lower full triple value. \end{proof}

The margin condition is stronger than the old hard-class condition: the reference strategy must beat all bad stochastic encoders in $\mathcal Q$, not only bad deterministic partitions.

5. Marginal fidelity and the small-complexity limit

For any router-generator pair, define the marginal conditional law

$$m_{r,g}(y\mid x) := \sum_z r(z\mid x)g(y\mid x,z).$$

\begin{proof} For fixed $x$, let

$$P_x(y,z)=p(y\mid x)q(z\mid x,y)=q(z\mid x)q(y\mid x,z)$$

and

$$M_x(y,z)=r(z\mid x)g(y\mid x,z).$$

The $Y$-marginals are $p(\cdot\mid x)$ and $m_{r,g}(\cdot\mid x)$. Data processing gives

$$\operatorname{KL}(p(\cdot\mid x)\|m_{r,g}(\cdot\mid x)) \le \operatorname{KL}(P_x\|M_x).$$

Factoring $P_x$ and $M_x$ gives

$$\operatorname{KL}(P_x\|M_x) = \operatorname{KL}(q(\cdot\mid x)\|r(\cdot\mid x)) + \mathbb E_{Z\sim q(\cdot\mid x)} \operatorname{KL}(q(\cdot\mid x,Z)\|g(\cdot\mid x,Z)).$$

Averaging over $X$ proves the claim. \end{proof}

This makes the finite-$\lambda$ tradeoff explicit. The profiled optimizer may choose a simpler approximate router-generator pair, incurring nonzero prior or decoder gaps. Exact marginal factorization is therefore not guaranteed at finite $\lambda$.

The high-fidelity limit is simple. Fix $q$ and $\beta>0$. If the exact induced pair

$$r_q(z\mid x)=q(z\mid x), \qquad g_q(y\mid x,z)=q(y\mid x,z)$$

belongs to $\mathcal R\times\mathcal G$ and has finite $C(r_q,g_q)$, then any minimizer $(r_\lambda,g_\lambda)$ of $A_{\beta,\lambda}(q)$ satisfies

$$\operatorname{DecGap}(q,g_\lambda) + \beta\operatorname{PriorGap}(q,r_\lambda) \le \lambda C(r_q,g_q).$$

Thus

$$\operatorname{DecGap}(q,g_\lambda)\to0, \qquad \operatorname{PriorGap}(q,r_\lambda)\to0$$

as $\lambda\downarrow0$, and the marginal law $m_{r_\lambda,g_\lambda}(y\mid x)$ approaches $p(y\mid x)$ in expected KL.

Interpretation: $\lambda\to0$ is the high-fidelity, weak-implicit-bias limit. It restores marginal fidelity under exact realizability, but it also weakens semantic selection because the encoder score approaches maximum conditional information.

6. Deterministic hard-strategy specialization

Let

$$\Phi_K=\{\phi:\mathcal X\times\mathcal Y\to[K]\}.$$

For $\phi\in\Phi_K$, define the hard encoder

$$q_\phi(z\mid x,y)=\mathbf 1\{z=\phi(x,y)\}.$$

Let $Z_\phi=\phi(X,Y)$. The induced router and generator are

$$r_\phi(z\mid x)=\mathbb P(Z_\phi=z\mid X=x),$$

and

$$g_\phi(y\mid x,z)=p(y\mid x,Z_\phi=z)$$

on positive-probability branches. Then

$$p(y\mid x)=\sum_z r_\phi(z\mid x)g_\phi(y\mid x,z).$$

Also,

$$I_{q_\phi}(Y;Z\mid X)=I(Y;Z_\phi\mid X)=H(Z_\phi\mid X).$$

Define $C(\phi):=C(r_\phi,g_\phi)$. The deterministic profiled score is

$$J^{\mathrm{prof}}_{\beta,\lambda}(\phi) = I(Y;Z_\phi\mid X) - \frac{A_{\beta,\lambda}(q_\phi)}{1-\beta}.$$

If the exact induced pair wins the inner profile,

$$A_{\beta,\lambda}(q_\phi)=\lambda C(r_\phi,g_\phi),$$

then

$$J^{\mathrm{prof}}_{\beta,\lambda}(\phi) = I(Y;Z_\phi\mid X) - \frac{\lambda}{1-\beta}C(\phi).$$

This recovers the old information-complexity score

$$J_\alpha(\phi)=I(Y;Z_\phi\mid X)-\alpha C(\phi), \qquad \alpha=\frac{\lambda}{1-\beta}.$$

So the original hard-class theorem is best read as the zero-gap specialization of the expressive profiled ELBO theorem.

7. Empirical recovery

The same margin controls empirical recovery. Let $\widehat{\bar{\mathcal L}}_n$ be an empirical profiled loss. If

\sup_{q\in\mathcal Q} \left| \widehat{\bar{\mathcal L}}_{\beta,\lambda,n}(q) - \bar{\mathcal L}_{\beta,\lambda}(q) \right| < \frac{(1-\beta)\Delta_{\beta,\lambda}^{\rho,\mathrm{prof}}(\delta)}{2},

then every empirical profiled minimizer satisfies

d_\rho(\widehat q_n,S^\star)<\delta.

For a finite encoder class $|\mathcal Q|=N$, if each centered empirical profiled loss is sub-Gaussian with variance proxy $\sigma^2$, then a union bound gives the sufficient condition

$$n \ge \frac{8\sigma^2} {(1-\beta)^2(\Delta_{\beta,\lambda}^{\rho,\mathrm{prof}}(\delta))^2} \log\frac{2N}{\eta}.$$

The important scaling is

$$n \asymp \frac{\text{statistical complexity of the encoder class}} {(1-\beta)^2(\text{profiled recovery margin})^2}.$$

8. Interpretation and limits

The clean paper-level message is:

1. Low-$\beta$ ELBO supplies information pressure. For \beta<1, the objective rewards latents that capture residual variation in $Y$ after conditioning on $X$.
2. Profiled router-generator complexity supplies selection. $A_{\beta,\lambda}(q)$ penalizes encoders whose induced routing law and branch conditionals are hard to realize faithfully with a simple router-generator pair.
3. Finite $\lambda$ can trade fidelity for simplicity. Exact preservation of $p(y\mid x)$ is guaranteed by zero variational gaps, not by the ELBO objective alone.
4. The limit $\lambda\to0$ restores high fidelity under realizability. For fixed $q$ and $\beta>0$, the exact induced pair becomes competitive enough to force the profiled gaps to zero. The cost is that semantic selection weakens and the encoder objective tends toward maximum information.
5. The deterministic $\Phi_K$ result remains important. It is the clean hard-strategy partition special case, but it should not be presented as the main expressive ELBO theorem.

The margin condition is still an argmax-style assumption: recovery occurs when the reference strategy beats every bad competitor under the profiled score. Its value is not that the condition is automatic, but that it identifies the exact selection pressure needed and gives the same gap that controls empirical recovery.