notation_and_terminology

Notation & Terminology

Notation

• $\mathcal{V}$: vocabulary / token space
• $x$: input instance; $x = (x_1, \ldots, x_t) \in \mathcal{X} \subset \mathcal{V}^*$
• $y$: solution or response; $y = (y_1, \ldots, y_t) \in \mathcal{Y} \subset \mathcal{V}^*$
• $z$: latent (strategy) variable; $z \in \mathcal{Z}$; $\mathcal{Z}$ can be a continuous vector space $\mathbb{R}^{d_{latent}}$ or a discrete space.
• $p(y|x)$ (autoregressive) language model, $p(y_1, \ldots, y_T | x) = \prod_t p(y_t | x, y_{<t})$
• $r(z|x)$: (strategy) router
• $g(y|x,z)$: strategy-conditioned generator

Terminology decisions (tentative; to be finalized during writing)

Core object:

• latent-variable router-generator factorization: $(r(z|x), g(y|x,z))$ is a latent-variable factorization of $p(y|x)$ if $\int_z r(z|x) g(y|x,z) d\mu(z) = p(y|x)$.
• Use "factorization" as the primary noun. "Disentangled" is used adjectivally — a disentangled factorization — to refer to the goal property, not as a standalone noun. Avoid "disentanglement" as the primary object of study to reduce confusion with the β-VAE/FactorVAE disentanglement literature.

Latent variable $z$:

• Technical term: "strategy" or "(latent) strategy variable" or "latent (strategy) variable".
• Reserve "mode" for the more intuitive distributional-mode framing in intro/abstract only (e.g. "modes of reasoning"). Avoid "mode" in methodology/experiments sections — overloaded (distribution mode vs. reasoning mode).

Component names:

• $r(z|x)$: router (or strategy router)
• $g(y|x,z)$: strategy-conditioned generator (or just generator when context is clear)

Setting term:

• Use "adaptation" or "factorization of a pretrained model".
• Avoid post-training (overloaded with RLHF/SFT) and distillation (overlapping but misleading).

Scalar quantity: residual surprisal under the $z$-free model: $-\log p(y|x,z) - (-\log p(y|x))$ (or the ratio $-\log p(y|x,z) / -\log p(y|x)$ for the normalized form). Used to describe "what $z$ contributes beyond $x$" as a measurable quantity.

• Preferred gloss for the difference form: code-length reduction / absolute information gain.
• Preferred gloss for the ratio form: fractional information gain / leftover fraction of baseline surprisal.
• Preferred explanatory question: "what fraction of the residual surprise in $y$ given $x$ is explained by $z$?"

Methodology objective family (tentative; to be finalized):

• Umbrella name: base-model-relative excess reconstruction (short form: excess reconstruction).
• Rationale: "base-model-relative" names what "excess" is measured against — the pretrained $z$-free baseline $p(y|x)$ — which is what makes the setting unique.
• Sub-variants (see CVAE objective and loss terms):
  • globally base-model-relative reconstruction (baseline-normalized scale restoration)
  • example-level base-model-relative reconstruction (per-example inverse-surprisal normalization)
  • token-weighted base-model-relative reconstruction (per-token weighting by base-model surprisal)
• Alternative umbrella held in reserve: residual-surprisal reconstruction — tighter tie to the scalar quantity above and to information-theoretic framings; revisit once theory component's role is clearer.
• Useful interpretive glosses that should remain explanatory, not renaming:
  • baseline-normalized reconstruction as a global rate-distortion rescaling
  • example-level normalization as a heterogeneous $\beta$-VAE / example-dependent latent budget
  • example-level normalization as an equalization force, not a preference for "easy" examples

Open items

• Finalize the methodology umbrella term (base-model-relative excess vs. residual-surprisal) after section arc / theory scope decisions.
• Decide on display names for ablation components used in figures and tables — these may be shorter than the full descriptive forms above.
• Avoided shorthand: do not summarize the example-level objective as "giving more latent budget to easy examples." Prefer "normalize away absolute difficulty so high-surprisal examples do not dominate by raw loss mass."