Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators

Computer Science > Artificial Intelligence [Submitted on 5 Jun 2026] Safety is Contextual, LLM-Judges Are Not: Navigating the Rigid Priors of Evaluators Anissa Alloula, Federico Licini, Ava Batchkala, Seraphina Goldfarb-Tarrant LLMs-as-judges are the only way to evaluate safety at scale. Despite their importance, LLM-judges themselves are rarely evaluated beyond human agreement in simple, static benchmarks. We therefore investigate two under-explored but crucial properties of LLMs-as-judges: their susceptibility to relying on in context-information, and their steerability to differing safety definitions, which may not align with their internal safety priors. We evaluate the safety judging abilities of many generalist LLMs and safety-specific judges, and investigate the impact of task demonstrations, novel in-context information, and changing safety definitions. We find that while LLM-judges can learn from new information, they are broadly unlikely to adjust their evaluations if the context or safety definition contradicts their prior. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.07874 [cs.AI]   (or arXiv:2606.07874v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.07874 Focus to learn more Submission history From: Anissa Alloula [view email] [v1] Fri, 5 Jun 2026 22:11:26 UTC (2,547 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)