Paper Theory Scope: What Theory Should We Include?

This is a living planning note tracking decisions and implications about the theory component of the paper. It is maintained during storyboarding and refined as we develop the paper.

Related notes:

• Story boarding
• Paper scope and framing
• Problem framing
• Formalizing strategy identification
• Toy model
• Identifiability subarea

Paper framing commitment

The paper is committed to the positioning: "We formulate a new problem (latent-variable factorization of pretrained LMs) and initiate its study." Under this framing, the claim is not that the problem is solved, but that we identify the core challenges unique to the setting, develop methodology that addresses them, and provide scaffolding for future methodology to build on.

Implication: a theoretical component is highly appropriate for this framing.

What theory is already available

Existing theory assets, in order of maturity:

• Formalization of the factorization problem (02): mixing operator $M(r,g)$, factorization set $\mathcal{F}(p)$, existence/identification/selection trichotomy, formal definitions of generator-class expressiveness (absorption, approximation-richness), equivalence quotienting, composite selection criterion (informativeness + complexity + separation). Near-publishable prose quality. Contains non-identifiability impossibility argument.
• Toy model seed (03): mixture-over-deterministic-functions setup, oracle identification question $\max I(z;y)$ subject to factorization constraint. No theorems yet.
• Identifiability core questions and preliminary material (identifiability subarea): more exploratory; not yet aligned with the strategy-identification formalization.

Planned placement in the paper

Theoretical components can be incorporated across three sections:

1. In Problem Statement section: formal definition of the factorization problem — mixing operator, factorization set, the three-question (existence / identification / selection) trichotomy. Load-bearing for Option A framing.
2. In Methodology motivation: non-identifiability result under expressive generators, and the composite selection-criterion framing that motivates the modified objective.
3. In dedicated Theory section (after Experiments, marked placeholder): toy-model recovery result and any sample-complexity or objective-consistency results that mature in time. Presence is conditional on development progress.

Implications for what theory to develop

Decisions and reasoning accumulated during storyboarding:

• Recovery-style result on the toy model is the highest-value target. A theorem of the form "under conditions C, selection criterion J applied to this toy model recovers the ground-truth strategies up to permutation" would convert the paper from "formulation + heuristic methodology" to "formulation + principled methodology with initial recovery guarantee on a toy model." This is the single largest lift available.
• Non-identifiability impossibility is already a viable theoretical contribution even without a recovery theorem. Framing it as motivation for why methodology must introduce inductive bias is compelling and matches the paper's posture.
• Avoid over-engineering the theory. A short, clean recovery result on a simple toy model beats a complicated one on a more realistic setting. Reviewers reading a "new problem" paper do not expect definitive theorems — they expect evidence that the problem is tractable and well-posed.
• Theory should connect back to the methodology, not stand alone. The selection criterion used in theorems should mirror (or at least morally correspond to) the training objective studied empirically. Disconnected theory weakens both halves.

Open questions

• Do we have time / inclination to prove a recovery theorem on the toy model? If not, is the formalization + impossibility result alone compelling enough for the marked Theory section to exist?
• Should the toy-model analysis be purely oracle ($\max I(z;y)$ subject to the factorization constraint) or also analyze the actual training objective we use empirically?
• How much of the composite selection criterion ($I(Z;Y|X)$ + complexity + separation) should be exposed in the paper's theory versus kept as motivation for the methodology?
• Does the identifiability subarea have anything reusable, or is it superseded by the strategy-identification formalization?

Status

Seed note. Will be updated as storyboarding proceeds and as theory development produces concrete results or decisions.