On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models

Citation

Authors: Charlie Zhang, Graham Neubig, and Xiang Yue
Year: 2025
Venue: arXiv preprint
arXiv: arXiv:2512.07783
URL: https://arxiv.org/abs/2512.07783

One-Line Takeaway

In a controlled synthetic setting, RL extends reasoning only when the base model has enough unused headroom and the RL data sit at the model's edge of competence; otherwise RL mainly sharpens or stalls.

Technical Core

The paper uses contamination-free synthetic reasoning tasks with explicit DAG structure so it can separately vary depth, context, pre-training exposure, and RL data.
Two generalization axes are studied: extrapolative depth generalization and contextual transfer across different surface templates with shared logic.
The core conclusion is conditional: RL can produce real capability gains, but only when pre-training has supplied the right primitives and the post-training curriculum is calibrated to what the model can almost, but not yet fully, do.

Method

Train small decoder-only models from scratch on synthetic reasoning corpora.
Insert an optional mid-training stage as a bridge between broad pre-training and RL.
Evaluate how different allocations of compute across pre-training, mid-training, and RL affect pass rates on in-distribution, edge-of-competence, and harder out-of-distribution tasks.

Empirical Evaluation

Sparse exposure during pre-training can be enough to seed contextual generalization, after which RL can strengthen transfer.
Mid-training is a strong but under-discussed lever: it establishes priors that let later RL work more effectively.
Process-aware rewards improve fidelity and reduce reward hacking relative to pure outcome rewards.

Assumptions And Limits

The strongest claims come from a tightly controlled synthetic domain rather than from frontier LLM benchmarks. The value of the paper is causal clarity more than immediate benchmark realism.

Relevance To This Project

This paper maps directly onto the repo's exploration hypothesis. It suggests that whether RL can reveal new strategies depends heavily on whether the relevant primitives are already represented and whether the training distribution reaches the model's near-frontier rather than sitting either far below or far above it. That is conceptually close to the repo's concern about router support and reachable latent strategies.

Comparison Notes

The Coverage Principle gives a complementary pre-training view by asking how coverage predicts later post-training success. Yue et al. (2025) and Wu et al. (2025) emphasize the support-constrained failure mode when those enabling conditions are absent. This paper is useful because it specifies when the pessimistic story can fail.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:2512.07783; no local cache or downloaded source artifacts were created.

Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.