MMR-GRPO: Accelerating GRPO-Style Training through Diversity-Aware Reward Reweighting

Citation

• Authors: Kangda Wei and Ruihong Huang
• Year: 2026
• Venue: arXiv preprint
• arXiv: arXiv:2601.09085
• URL: https://arxiv.org/abs/2601.09085

One-Line Takeaway

MMR-GRPO keeps the standard GRPO structure but reweights completions by a diversity-aware MMR score so that redundant samples contribute less and informative groups train faster.

Technical Core

• Standard GRPO uses only group-relative reward statistics and ignores semantic redundancy among completions.
• MMR-GRPO embeds each completion, computes pairwise similarity, and replaces raw rewards with an MMR-style score that balances quality against dissimilarity from already selected responses.
• The diversity-relevance tradeoff parameter is made adaptive: when raw rewards are tightly clustered, the method leans more heavily on diversity; when rewards are spread out, it leans more heavily on relevance.

Method

• Embed completions with a sentence encoder.
• Build a greedy MMR ordering from the current reward group.
• Use the reweighted scores in place of raw rewards when computing GRPO-style advantages.

Empirical Evaluation

• The method reaches comparable peak performance with far fewer training steps and much lower wall-clock time than vanilla GRPO-style baselines.
• It preserves pass@k behavior, so the faster convergence does not come from crippling inference-time exploration.

Assumptions And Limits

This paper improves training efficiency and within-group diversity, but it is not primarily a paper about capability expansion beyond base-model support. Its gains can coexist with the broader support-shrinkage story diagnosed elsewhere.

Relevance To This Project

The main relevance is diagnostic. If an RLVR recipe improves after diversity-aware reweighting, that does not automatically mean the model is learning new latent strategies; it may only mean the training signal inside the already-sampled solution set became more informative. That distinction matters for how the repo should interpret RL improvements.

Comparison Notes

HA-DW also modifies GRPO-style weighting, but it focuses on prompt difficulty bias rather than semantic redundancy. BroRL broadens exploration by increasing rollout count, whereas MMR-GRPO extracts more value from a fixed-size group. Compared with RepExp, this method is conservative: it reweights sampled completions instead of trying to search for genuinely novel ones.

Related Notes

• Your Group-Relative Advantage Is Biased
• BRORL: Scaling Reinforcement Learning via Broadened Exploration
• Reinforcement Learning via Self-Distillation