BRORL: Scaling Reinforcement Learning via Broadened Exploration

Citation

Authors: Jian Hu, Mingjie Liu, Ximing Lu, Shizhe Diao, Fang Wu, Yejin Choi, Pavlo Molchanov, Jun Yang, Jan Kautz, Yi Dong, and Zaid Harchaoui
Year: 2025
Venue: arXiv preprint
arXiv: arXiv:2510.01180
URL: https://arxiv.org/abs/2510.01180

One-Line Takeaway

BroRL argues that one important RLVR bottleneck is insufficient exploration breadth per prompt, and that massively increasing rollout count can revive models that have already plateaued under standard prolonged RL.

Technical Core

The paper analyzes a one-step RLVR update through a mass-balance decomposition of correct-token probability mass.
The unstable piece is an unsampled coupling term whose harmful effect shrinks as the number of rollouts per example N grows.
This motivates a new scaling axis for RLVR: broaden exploration by greatly increasing N, instead of only increasing the number of optimization steps.

Method

Start from a PPO-style RLVR recipe with dynamic sampling and related engineering from the ProRL line of work.
Increase rollouts per prompt from a small default value to hundreds, and scale the learning rate accordingly.
Use this high-N regime to make gradient updates depend less on poorly sampled tails and more on well-observed variation.

Empirical Evaluation

In both toy simulations and LLM experiments, larger N reduces knowledge shrinkage and improves stability.
On a plateaued 1.5B reasoning model, BroRL continues to gain where the baseline prolonged-RL recipe saturates or degrades.

Assumptions And Limits

The method is expensive in per-step generation, and its practical value depends on whether hardware and system design can exploit the larger rollout batches efficiently enough to offset the extra sampling cost.

Relevance To This Project

BroRL is relevant because it isolates one concrete sense in which passive RLVR may fail to explore: the policy simply does not see enough diverse evidence per problem. The repo's strategy-disentanglement approach is more structural, but BroRL is a useful baseline because it asks how far one can go by broadening search without changing the model's latent representation.

Comparison Notes

Compared with RepExp, BroRL broadens exploration by brute-force rollout breadth rather than by representation-aware novelty bonuses. Compared with MMR-GRPO and HA-DW, it changes the sampling regime itself rather than fixing how a small sampled group is weighted.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:2510.01180; no local cache or downloaded source artifacts were created.

Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.