Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight

Computer Science > Artificial Intelligence [Submitted on 29 May 2026] Weak Critics Make Strong Learners: On-Policy Critique Distillation for Scalable Oversight Can Jin, Jiakang Li, Rui Wu, Eddy Zhang, Dimitris N. Metaxas As large language models become stronger, weak supervisors may fail to provide reliable labels, preferences, or final judgments for complex outputs, limiting both weak-to-strong generalization and scalable oversight. We study a more tractable form of weak supervision: using a weak model as a critic rather than as a labeler or judge. Instead of solving the task or selecting the correct answer, the weak critic only needs to provide a non-misleading revision direction that helps the strong model better use its own knowledge. We call this setting *weak-critic strong oversight*. We first show that weak critiques can improve frozen strong models at inference time, and that critique quality is key to this improvement. We then propose progressive on-policy critique distillation (**OPCD**), which filters high-quality critiques and distills critic-guided behavior into the strong model through adaptive self-teacher signals. Experiments on reasoning and alignment benchmarks show that our method improves strong models over training epochs, suggesting an effective path for scalable oversight with weak supervision. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.00424 [cs.AI]   (or arXiv:2606.00424v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.00424 Focus to learn more Submission history From: Can Jin [view email] [v1] Fri, 29 May 2026 23:21:48 UTC (102 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)