HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation

Computer Science > Artificial Intelligence [Submitted on 10 Jun 2026] HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation Haoran Liu, Yuwei Zhang, Xiyao Li, Bohan Lyu, Jingbo Shang Reinforcement learning typically improves multi-turn agent capabilities through the terminal outcome of the trajectories, which makes it difficult to determine credit assignments for each intermediate turns. Recent on-policy self-distillation methods offer a promising alternative by converting privileged feedback into dense token-level supervision through a self-teacher. Our study is motivated by the unexpected performance degradation observed when naively extending this paradigm to multi-turn settings, which we attribute to a lack of alignment between privileged feedback, such as successful trajectories or terminal outcomes, and the student's current decision context. We introduce HERO, a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback. After each rollout, HERO reflects on the completed interaction to convert each observation into a compact turn-level diagnosis, that captures actionable feedback about the original action such as its necessity, validity or failure cause. On TauBench and WebShop, HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO. It is especially effective under limited training turn budgets, where successful rollouts are rare and GRPO provides weak reward-contrast signals. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.11559 [cs.AI]   (or arXiv:2606.11559v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.11559 Focus to learn more Submission history From: Haoran Liu [view email] [v1] Wed, 10 Jun 2026 01:35:34 UTC (159 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)