Task Data And Evaluation Setup

This note records paper-facing language for the experimental task/data setup: what to call the benchmark, how to describe $x$ and $y$, how to distinguish the original task/data distribution from the frozen base-model response distribution, and how to describe strategy labels and inferred strategy attribution.

Benchmark Name

Use controlled multi-strategy algorithmic benchmark at first mention.

Use algorithmic strategy benchmark as the shorter recurring name.

Avoid making multi-task multi-strategy algorithmic benchmark the primary phrase. It is accurate but repetitive, and it makes the name harder to scan in prose and captions.

Reference phrasing:

We evaluate on a controlled multi-strategy algorithmic benchmark: a suite of task families in which each problem instance admits multiple valid solution strategies with known strategy annotations.

Short version:

The algorithmic strategy benchmark gives controlled access to problem instances, solution traces, and reference strategy labels.

Benchmark Motivation

The benchmark exists to make strategy-aligned factorization measurable. The paper's methodology aims to learn a router-generator factorization of a pretrained language model, but in open-ended language modeling there is no direct ground truth for whether the learned latent has recovered strategy structure. Controlled tasks provide reference strategy annotations, making it possible to evaluate whether the latent aligns with intended strategies rather than arbitrary predictive features, and whether its semantics remain consistent across related inputs.

Reference phrasing:

To evaluate whether a method can learn strategy-aligned factorizations of a pretrained language model, we need a setting where the relevant strategy structure is observable. The controlled multi-strategy algorithmic benchmark provides this: solution traces carry reference strategy annotations, enabling precise measurement of strategy alignment and cross-input semantic consistency.

Here cross-input semantic consistency is the preferred paper term for the global semantic consistency of the latent space across related problem instances.

Use reference strategy annotations or ground-truth strategy labels when emphasizing the evaluation object. Reference strategy annotations is slightly softer and avoids implying that the labels are a unique ontology of all possible strategies; ground-truth strategy labels is appropriate when discussing controlled-task metrics.

Experimental Methodology

Describe the experimental procedure as a two-stage simulation of a pretrained model with an entangled strategy distribution.

1. Fit a language model to the task/data distribution. This may be a from-scratch model or a pretrained backbone such as Qwen adapted to the benchmark. The resulting base model has learned a mixed response distribution over multiple valid solution strategies.
2. Apply the router-generator methodology to the frozen base model, learning a factorization of its strategy-entangled response distribution.

Reference phrasing:

Our experimental protocol first fits a language model to the benchmark task distribution, producing a base model whose conditional response distribution mixes several valid solution strategies. We then freeze this base model and train a router-generator factorization with respect to its response distribution $p_\theta(y|x)$.

Another useful version:

The benchmark simulates the relevant pretrained-model setting in controlled form: the base model is trained on a distribution containing multiple reference strategies per task, and the factorization method must recover a strategy-aligned decomposition of the resulting mixed strategy distribution.

Use strategy-entangled language model or strategy-entangled base model for the model after it has been fit to the task distribution and before factorization.

Core Objects

Use problem instance as the default prose term for $x$ in the experiments section. Use input in equations and compact technical descriptions.

Use solution trace as the default prose term for $y$ in the experiments section. Use response sequence or response trace when connecting back to the general language-model formulation.

Use task family for the individual algorithmic tasks. This is clearer than subtask in paper prose, even though the implementation uses subtask_name inside the aggregate multi-task wrapper.

Use strategy or solution strategy for the high-level procedure instantiated by a solution trace. At first mention in the experiments section, solution strategy is useful because it anchors the term to the task output.

Reference phrasing:

Each example consists of a problem instance $x$ and a solution trace $y$. A solution trace may instantiate one of several valid solution strategies for the same problem instance.

General-language-model bridge:

In the general formulation, $y$ is a response sequence. In the controlled benchmark, responses are solution traces whose strategy can be parsed and attributed.

Distribution Language

Distinguish the original task/data distribution from the frozen base-model distribution used for factorization.

Use task/data distribution or benchmark distribution for the distribution $\mathcal D$ over problem instances and reference solution traces used to fit the base model.

Use base-model response distribution for $p_\theta(y|x)$, the conditional distribution defined by the frozen base model after fitting.

The paper can describe base-model fitting as:

$$(x,y)\sim\mathcal D$$

and factorization training as:

$$x\sim\mathcal D_X,\qquad y\sim p_\theta(\cdot|x).$$

Use $\mathcal D_X$ for the marginal distribution over problem instances when the response is sampled from the base model rather than drawn from the original task/data distribution.

Reference phrasing:

The task/data distribution $\mathcal D$ is used to fit the base model $p_\theta$. The factorization stage then treats $p_\theta(y|x)$ as the object to be decomposed: inputs are drawn from the benchmark input distribution, while solution traces are sampled from the frozen base-model response distribution.

Short version:

The factorization is trained with respect to the base-model response distribution $p_\theta(y|x)$, not directly against strategy labels.

Avoid saying only train on the dataset when describing the factorization stage. That phrasing obscures the central point that the methodology is factorizing the frozen base model.

Strategy Labels And Attribution

Use strategy annotations or ground-truth strategy labels for labels provided by the controlled benchmark and used for evaluation.

Use inferred strategy or strategy attribution for the strategy recovered by parsing and validating a generated solution trace.

Reference phrasing:

Strategy annotations are withheld from factorization training and used only for evaluation.

Generated-sample phrasing:

For generated solution traces, we infer a strategy attribution by parsing the trace and matching it to the benchmark's reference strategy validators.

The paper usually does not need to foreground invalid, ambiguous, or off-strategy generated outputs unless discussing evaluation details. When needed, use uniquely attributed, ambiguous, and invalid as the compact descriptors.

Relationship To Neighboring Concepts

The definition of a strategy and the evaluation-only role of labels connects to Latent strategy variable.

The target property measured by the benchmark is strategy alignment and recovery.

Cross-input semantic consistency and Analogical Consistency are enabled by the benchmark's reference strategy annotations; see Cross-input semantic consistency.

The factorization trained on the frozen base-model distribution is a router-generator factorization.