Fragility of Value under Imperfect Alignment

Computer Science > Artificial Intelligence [Submitted on 30 Jul 2026] Fragility of Value under Imperfect Alignment Winter Cross As more responsibility is placed upon AI systems, it becomes increasingly important to guarantee that these systems are aligned with humanity. A common fear in AI safety is that human value is fragile -- that is, optimizing too heavily for an imperfect proxy to human values will lead to a catastrophic outcome. In this paper, we present a model of the alignment problem where an agent undergoes idealized alignment training that guarantees its value function satisfies a proxy condition before optimizing the world. Our primary results identify conditions on the human value function and the accuracy of several proxy conditions under which an agent with an η -catastrophic value function, one that is guaranteed to take the expectation of human value below η in the limit of optimizing power, would be deployed. Our results highlight the danger of overoptimization and motivate AI designs that limit optimization pressure, such as quantilizers, rather than relying solely on pre-deployment training. Comments: 24 pages, 7 figures Subjects: Artificial Intelligence (cs.AI) MSC classes: 68T01 ACM classes: I.2.0 Cite as: arXiv:2607.28881 [cs.AI]   (or arXiv:2607.28881v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2607.28881 Focus to learn more Submission history From: Winter Cross [view email] [v1] Thu, 30 Jul 2026 22:52:34 UTC (503 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-07 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)