CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Computer Science > Artificial Intelligence [Submitted on 12 Jun 2026] CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning Ayoub Belouadah, Sylvain Kubler, Yves Le Traon Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods Comments: Accepted as a Spotlight paper at the 43rd International Conference on Machine Learning (ICML 2026) Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.14415 [cs.AI]   (or arXiv:2606.14415v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.14415 Focus to learn more Submission history From: Ayoub Belouadah [view email] [v1] Fri, 12 Jun 2026 12:48:56 UTC (2,649 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)