The Invisible Leash? Why RLVR May or May Not Escape Its Origin
The Invisible Leash? Why RLVR May or May Not Escape Its Origin
Citation
- Authors: Fang Wu, Weihao Xuan, Ximing Lu, Mingjie Liu, Yi Dong, Zaid Harchaoui, and Yejin Choi
- Year: 2025
- Venue: arXiv preprint
- arXiv: arXiv:2507.14843
- URL: https://arxiv.org/abs/2507.14843
One-Line Takeaway
This paper formalizes why current RLVR often remains tethered to the base model's initial support: it preserves and sharpens already-accessible solutions much more reliably than it discovers new ones.
Technical Core
- The key object is empirical support: correct completions whose probability is high enough to be practically reachable under finite sampling.
- The paper partitions correct solutions into preservation, expansion, shrinkage, and out-of-support categories, then summarizes support dynamics with metrics such as support retention rate, net discovery rate, support dynamic score, and net support change rate.
- Across benchmarks, RLVR shows high retention but very low discovery, and shrinkage often exceeds expansion, which explains why RLVR may improve
pass@1while hurting high-budgetpass@k.
Theoretical Result
- RLVR is analyzed as support-constrained optimization: standard on-policy updates cannot place meaningful mass on solutions that are truly absent from the base distribution.
- The paper also proves an entropy-reduction tendency, which clarifies why local stochasticity can increase while global answer diversity still shrinks.
Assumptions And Limits
The analysis is built around empirical accessibility under finite sampling, so it is still a functional notion of capability rather than a full mechanistic theory of internal reasoning representations.
Relevance To This Project
This note is central for the repo's framing. It provides a language for saying that the problem is not only "RL does not generalize enough," but more specifically "RLVR often contracts the accessible support over reasoning traces." That is exactly the failure mode a latent-strategy factorization is trying to address.
Comparison Notes
Yue et al. (2025) reaches a similar conclusion through pass@k and perplexity analysis; this paper adds a sharper support-level formalism. The Sharpening Mechanism is philosophically adjacent because it explains why self-improvement often looks like redistributing mass inside an existing support. The Coverage Principle gives a complementary pre-training story: if support coverage matters for post-training success, then RLVR's shrinkage behavior is especially consequential.
Retrieval: alphaXiv MCP get_paper_content on arXiv:2507.14843; no local cache or downloaded source artifacts were created.
Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.
