Reinforcement Learning via Self-Distillation
Reinforcement Learning via Self-Distillation
Citation
- Authors: Jonas Hubotter, Frederike Lubeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Thomas Kleine Buening, Idan Shenfeld, Carlos Guestrin, and Andreas Krause
- Year: 2026
- Venue: arXiv preprint
- arXiv: arXiv:2601.20802
- URL: https://arxiv.org/abs/2601.20802
One-Line Takeaway
SDPO treats the current model, conditioned on rich feedback, as a self-teacher that can assign dense token-level credit to its own failed rollouts.
Technical Core
- The paper formalizes reinforcement learning with rich feedback (RLRF): environments often return textual error information, failing tests, or other structured feedback instead of only a scalar success bit.
- Given prompt
x, failed rollouty, and feedbackf, the self-teacher ispi_theta(. | x, f). SDPO trains the student policy to match this feedback-conditioned teacher on the original attempt. - The resulting loss acts like dense policy-gradient credit assignment: instead of one scalar advantage per sequence, the model gets token-level information about what should shift after seeing the feedback.
Method
- Sample rollouts from the current policy.
- Obtain rich feedback from the environment, or synthesize it in standard RLVR by using successful batch rollouts as feedback for failed ones.
- Minimize a logit-level distillation loss between the student and the stop-gradient self-teacher, with regularization to keep the teacher stable.
Empirical Evaluation
- SDPO improves sample efficiency and final performance in scientific QA, tool use, competitive programming, and hard test-time adaptation settings.
- A notable qualitative result is that SDPO often produces shorter and cleaner reasoning chains than GRPO-style baselines, suggesting improved credit assignment rather than mere verbosity inflation.
Assumptions And Limits
The method needs informative feedback or at least some successful trajectories that can be recycled as implicit feedback. It is therefore not by itself a solution to the pure zero-reward exploration problem.
Relevance To This Project
This paper is relevant less as a direct exploration method and more as a reminder that some apparent reasoning gains come from better credit assignment rather than from broader strategy discovery. For the repo, that distinction matters: disentangling latent strategies is a different objective from simply making already-supported trajectories easier to reinforce.
Comparison Notes
POPE and SOAR tackle how to obtain useful training signals on very hard problems. SDPO instead assumes richer feedback is available and then uses self-distillation to exploit it efficiently. Compared with MMR-GRPO and HA-DW, SDPO changes the granularity of supervision rather than reweighting sequence-level rewards.
Retrieval: alphaXiv MCP get_paper_content on arXiv:2601.20802; no local cache or downloaded source artifacts were created.
Verified from source: verification in this ingestion pass was limited to the alphaXiv whole-paper report and arXiv metadata exposed through that report. I did not run a separate targeted PDF query for this note.
