Synthetic Sequences Entangled Baseline

The first stage of the synthetic-sequences workflow trains an ordinary conditional language model on synthetic solution traces without an explicit latent strategy variable. The result is an entangled model $p_\theta(y \mid x)$ whose output distribution can mix multiple valid strategies for the same input.

Role In The Overall Experiment

This stage creates the source distribution that the disentangling stage later tries to refactor. The key empirical question is not whether the model can solve the tasks, but whether a strong mixed strategy model can later be decomposed into a controllable latent structure.

Training Regimes

The baseline trainer supports two broad regimes:

• random-initialized synthetic models with configurable model size,
• pretrained-backed models adapted to the synthetic tasks with either full fine-tuning or LoRA.

The repo-side trainer contract is documented in path:experiments/synthetic_sequences/entangled/README.md.

What Makes The Baseline "Entangled"

For a fixed input $x$, several strategies can produce distinct valid traces with the same final answer. Training on those traces directly produces a model that can represent the mixture in its hidden states and output distribution, but without a named variable that corresponds to strategy.

In that sense, strategy is present operationally but not factorized:

• the model can often generate multiple styles of solution,
• but there is no explicit $z$ that can be routed, intervened on, or evaluated directly.

Outputs Used Downstream

The baseline stage produces:

• trained model exports,
• sampled generation metrics,
• optional probe diagnostics,
• and run artifacts that can be loaded by the disentangling trainer.

Those artifacts support two downstream uses:

• initialize disentangling from an entangled run,
• or generate cached supervision rows from the entangled model's own output distribution.