Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Citation

Authors: Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang
Year: 2025
Venue: arXiv preprint
arXiv: arXiv:2504.13837
URL: https://arxiv.org/abs/2504.13837

One-Line Takeaway

The paper argues that current RLVR usually improves how efficiently a model samples reasoning paths it already had, but does not substantially expand the set of problems the base model can solve at sufficiently large sampling budgets.

Technical Core

The main object is the model's reasoning boundary, measured by pass@k: a problem counts as solvable if at least one of k samples is correct.
Across math, code, and visual reasoning tasks, RLVR improves pass@1 but often loses to the base model at large k, implying sharper local precision together with narrower global support.
Perplexity analysis shows that RLVR-generated successful traces are typically low-perplexity under the base model, suggesting that RLVR mostly reweights already-supported trajectories instead of discovering qualitatively new ones.

Method

Compare pretrained or instruction-tuned base models against RLVR-trained descendants across multiple model families and RL algorithms.
Track pass@k curves, solvable-problem overlap, accuracy histograms, and perplexity of RL-generated responses under the base model.
Contrast RLVR with distillation, which can inject traces from a stronger teacher and therefore genuinely broaden accessible reasoning patterns.

Empirical Evaluation

The central pattern is consistent: RLVR raises small-k performance, but the base model recovers more solutions when sampling is large enough.
Distillation behaves differently from RLVR: its pass@k curve can stay above the base model across the full budget range, which the paper treats as evidence of true boundary expansion.

Assumptions And Limits

The paper's notion of capability is tied to finite-sampling accessibility rather than to internal representation changes. It therefore says most about reachable solution support under current decoding and RL pipelines, not about every possible latent effect of RL.

Relevance To This Project

This is one of the clearest motivation papers for the repo. Its core claim matches the project's premise that passive on-policy RLVR tends to sharpen already-supported reasoning rather than discover new latent strategies. That makes latent-variable factorization and explicit strategy-level interventions look like a more natural route to structured exploration.

Comparison Notes

This paper is closely aligned with Wu et al. (2025), which recasts the same phenomenon in terms of empirical support and entropy reduction. Huang et al. (2024) gives a complementary theoretical picture in which self-improvement acts by sharpening high-likelihood regions of an existing distribution, while Tuyls et al. (2025) is explicitly trying to break this sharpening-only regime with deliberate exploration bonuses.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:2504.13837; no local cache or downloaded source artifacts were created.

Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.