Your Group-Relative Advantage Is Biased

Citation

Authors: Fengkai Yang, Zherui Chen, Xiaohan Wang, Xiaodong Lu, Jiajun Chai, Guojun Yin, Wei Lin, Shuai Ma, Fuzhen Zhuang, Deqing Wang, Jianxin Li, Yikun Ban, and Yaodong Yang
Year: 2026
Venue: arXiv preprint
arXiv: arXiv:2601.08521
URL: https://arxiv.org/abs/2601.08521

One-Line Takeaway

The paper shows that group-relative advantage estimates in GRPO-style RLVR are systematically biased, underweighting hard prompts and overweighting easy ones, then proposes a history-aware correction.

Technical Core

Theoretical analysis shows that the group-relative advantage estimator is biased relative to the true expected advantage.
When prompt success probability is below 0.5, hard prompts are underestimated; when it is above 0.5, easy prompts are overestimated.
This creates an undesirable learning dynamic: the policy over-exploits already-solved cases and under-learns from hard problems where exploration is most needed.

Method

HA-DW tracks an evolving estimate of model capability across batches using a history-aware difficulty anchor.
Current-group advantages are then reweighted as a function of how the prompt's observed difficulty deviates from that running capability estimate.
The intended effect is to amplify learning on genuinely hard prompts while damping overconfident updates on easy ones.

Empirical Evaluation

HA-DW improves GRPO, GSPO, and DAPO-style training across several math benchmarks and model sizes.
The biggest relative gains show up on hard prompts, which is consistent with the theory that standard group-relative weighting systematically neglects them.

Assumptions And Limits

This is a correction to the optimization geometry of group-based RLVR, not a full answer to exploration. If the base model never generates useful traces for a hard prompt, unbiased weighting alone may still be insufficient.

Relevance To This Project

The paper is relevant because it sharpens the distinction between two failure modes: support failure and weighting failure. A model may fail to discover new strategies either because the strategies are absent from reachable support, or because the RL signal underweights the hard prompts where those strategies would matter. The repo's exploration story probably needs to keep both mechanisms separate.

Comparison Notes

MMR-GRPO fixes redundancy inside sampled groups; HA-DW fixes bias across prompt difficulty. POPE and SOAR go after an earlier bottleneck by making hard problems reward-bearing in the first place. So this paper is best read as an optimization repair, not as a complete exploration method.

Source Note

Retrieval: alphaXiv MCP get_paper_content on arXiv:2601.08521; no local cache or downloaded source artifacts were created.

Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.