Metrics And Frontier Language

This note records paper-facing language for the core empirical metrics and plot-level claims. The guiding convention is to use self-explanatory property names in prose, figure titles, and claims, while defining the operational metrics that estimate those properties.

Strategy Alignment

Use strategy alignment as the umbrella property: the learned latent variable should correspond to meaningful solution strategies.

Use Router Strategy Decodability for the router-side linear-probe metric. This metric answers: can a simple probe recover strategy information from $Z \sim r_\phi(\cdot|X)$?

Use strategy decodability as the broader metric family for linear-probe measurements of strategy information in learned representations.

Use the following metric variants when additional precision is needed:

• router strategy decodability: strategy decodability measured from the router-side latent representation or router distribution.
• posterior strategy decodability: strategy decodability measured from the posterior-side latent representation or posterior distribution.

Avoid latent strategy decodability unless the evaluated representation is neither router nor posterior. It is too vague as a primary metric name; strategy decodability is the cleaner umbrella.

Avoid using Strategy Alignment as the scalar label for Router Strategy Decodability when the figure also reports a stronger cross-input metric. Router Strategy Decodability can be deceptively high when the latent contains input-local strategy information but does not induce reusable strategy semantics across related inputs.

Reference phrasing:

Router Strategy Decodability asks whether strategy information is linearly recoverable from router-sampled latents $Z \sim r_\phi(\cdot|X)$. This is a useful diagnostic for strategy alignment, but it is weaker than testing whether the same latent induces the same strategy across related inputs.

Use Router Strategy Decodability in captions, methods, and metric tables when naming this measurement. It is usually short enough for figure legends; if space is tight, use Router Decodability after the full term has been introduced.

Cross-Input Strategy Metrics

Use cross-input latent strategy alignment as a precise property name when emphasizing the stronger goal measured by the analogical metric: the latent should have strategy meaning that remains aligned across related inputs. This phrase is strong in prose but probably too long for compact figure legends.

Use analogical transfer for the intervention protocol: sample or infer a latent on a source input, reuse that latent for a related target input, and evaluate whether the induced strategy transfers.

Use Analogical Consistency for the scalar metric computed from that protocol. This is stronger than analogical agreement as a paper-facing term because it points to the desired property of the latent variable, while still being grounded in the agreement calculation used by the estimator.

Reference phrasing:

Strategy Alignment is measured as cross-input latent strategy alignment, using the Analogical Consistency metric under an analogical transfer intervention.

Compact caption phrasing:

In this figure, Strategy Alignment denotes cross-input latent strategy alignment, measured by Analogical Consistency.

Metric question:

This metric answers: does the same latent $Z$ induce the same strategy when transferred from $X$ to a related $X'$?

For compact figure legends:

• Strategy Alignment: default label for main paper figures that report the broader property and define the metric in the caption.
• Analogical Consistency: use when the figure is about the metric itself or when multiple strategy-alignment diagnostics are shown side by side.
• Router Strategy Decodability: use when the figure reports the weaker router-side linear-probe diagnostic.

Avoid Cross-Input Alignment as a standalone figure label because it does not say what is aligned. If space permits, Cross-Input Strategy Alignment is clearer, but it is likely too long for the main metric bars.

Current default for compact method metric bars: Strategy Alignment, with the caption defining it as cross-input latent strategy alignment measured by Analogical Consistency. This is acceptable only if Router Strategy Decodability is not also shown as a neighboring metric in the same legend.

Base-Relative Reconstruction

Use base-relative reconstruction as the default term for $\mathcal R_\theta$.

The per-example term is

$$\mathcal R_\theta(x,y;\phi,\xi).$$

Use the same symbol without explicit example arguments for its expectation over the base-model training distribution:

$$\mathcal R_\theta(\phi,\xi) := \mathbb E_{x\sim\mathcal D_X,\;y\sim p_\theta(\cdot|x)} \left[ \mathcal R_\theta(x,y;\phi,\xi) \right].$$

Do not introduce $\rho$ notation for this metric. Use $\mathcal R_\theta$ consistently.

Reported validation or test values are empirical estimates of $\mathcal R_\theta(\phi,\xi)$ and do not need separate notation in the main paper. Use phrases such as validation $\mathcal R_\theta$ when the split matters.

Reference phrasing:

Base-relative reconstruction $\mathcal R_\theta$ measures the fraction of z-free base-model surprisal remaining after conditioning on $z$.

Interpretations

Use fraction of baseline surprisal remaining as the main interpretation of $\mathcal R_\theta$.

Use residual surprise explained by $z$ for the complement:

$$1-\mathcal R_\theta.$$

Reference phrasing:

Base-relative reconstruction $\mathcal R_\theta$ measures the fraction of z-free base-model surprisal remaining after conditioning on $z$; its complement, $1-\mathcal R_\theta$, measures the residual surprise explained by $z$.

This complement is useful for plots because larger values are better and align visually with higher strategy alignment.

Figure-facing axis label:

Residual surprise explained by $z$

Caption definition:

Residual surprise explained by $z$ is $1-\mathcal R_\theta$, where $\mathcal R_\theta$ is base-relative reconstruction: the fraction of z-free base-model surprisal remaining after conditioning on $z$.

Frontier Language

Use alignment-reconstruction frontier when the figure or claim relates strategy alignment to reconstruction or residual-surprise quantities.

Use strategy-alignment frontier when the emphasis is the best achievable strategy alignment across methodology variants or hyperparameters.

Reference phrasing:

Runs trace an alignment-reconstruction frontier: models that explain more residual surprise with $z$ also learn more strategy-aligned latents.

Another useful phrasing:

The strategy-alignment frontier summarizes how much alignment different objective variants can obtain while preserving the base model's observable behavior.

Use operating regime for regions of hyperparameter behavior or method behavior, not as the main claim term.

Use Pareto-dominant variants only when the plot explicitly supports a Pareto comparison.

Collapsed Corner

Use collapsed corner for the region with low strategy alignment and low residual surprise explained by $z$, equivalently high $\mathcal R_\theta$.

Reference phrasing:

Standard ELBO lies in the collapsed corner: it preserves marginal fit while leaving $z$ inert.

Expanded version:

Standard ELBO lies in the collapsed corner of the alignment-reconstruction plot, with low strategy alignment and little residual surprise explained by $z$. This reflects marginal fit without a useful latent factorization.

Avoid saying that ELBO fails to model the data or fails to match $p_\theta(y|x)$. The point is that ELBO can preserve the observable conditional law while leaving the latent variable unaligned.

Relationship To Neighboring Concepts

Strategy alignment and strategy decodability connect to Strategy alignment and recovery.

Base-relative reconstruction and $\mathcal R_\theta$ connect to Base-directed reconstruction objective.

Cross-input metrics such as Analogical Consistency connect to Cross-input semantic consistency.

The ELBO collapsed-corner language connects to ELBO under-specification and the z-ignoring solution.

Terms That Need Care

Base-relative reconstruction ratio is acceptable when emphasizing that $\mathcal R_\theta$ is normalized by the z-free base-model reference scale, but the default term should be base-relative reconstruction.

Base-normalized reconstruction is understandable but less strong than base-relative reconstruction because it sounds like a scaling trick rather than a meaningful base-model-relative quantity.

Reconstruction ratio is too generic unless the ratio has already been defined.

Fractional reconstruction is catchy but ambiguous. Prefer using fraction of baseline surprisal remaining as an interpretive phrase.

Residual surprise explained by $z$ should refer to $1-\mathcal R_\theta$, not $\mathcal R_\theta$ itself.