Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR

Computer Science > Artificial Intelligence [Submitted on 4 Aug 2026] Don't Peek at the Answer: Outcome-Masked Group Relative Policy Optimization for Label-Free RLVR Yongshi Ye, Liang Zhang, Yidong Chen, Xiaodong Shi, Biao Fu Reinforcement Learning with Verifiable Rewards (RLVR) improves LLM reasoning but typically relies on ground-truth (GT) answers, limiting scalability. Voting-based label-free RLVR replace gold supervision with answer-level consensus from model samples. However, collapse arises when the same answer-level signal is used both to estimate rewards and to drive token-level policy optimization, encouraging the model to directly reinforce answer tokens rather than improve reasoning. We propose OM-GRPO, a label-free RLVR framework that decouples reward estimation from policy optimization. OM-GRPO masks gradients on the answer span while retaining answer-level rewards through a soft consensus signal, shifting optimization pressure away from answer tokens. We further introduce Contrast-Augmented Reward, which refines reward estimation via low-cost pairwise comparisons over existing trajectories without additional rollouts. Across diverse reasoning benchmarks and three LLM backbones, OM-GRPO consistently outperforms existing label-free RLVR methods and matches supervised GT-reward training with stable optimization. This stability is particularly beneficial in the Test-Time Training setting, where OM-GRPO surpasses majority voting by 4.24 points. Comments: 25 pages, 16 figures, and 9 tables Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2608.03119 [cs.AI]   (or arXiv:2608.03119v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2608.03119 Focus to learn more Submission history From: Yongshi Ye [view email] [v1] Tue, 4 Aug 2026 04:41:20 UTC (1,119 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)