Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy

Computer Science > Artificial Intelligence [Submitted on 6 Jun 2026] Stress-testing medical large language models reveals latent safety pathology beyond benchmark accuracy Yuan Shen, Xiaojun Wu, Linghua Yu Large language models (LLMs) are entering clinical practice based on benchmark accuracy that may fail to detect safety-relevant failure modes. Here we present AI-MASLD, a stress-audit framework that adapts the logic of metabolic stress testing from hepatology to the evaluation of clinical LLMs. Using 240 clinical cases across six narrative perturbation probes, we subjected seven models to double-stress testing and quantified performance through three indices: metabolic index (MI), perturbation flip rate (PFR), and counterfactual fairness index (CFI). Under clean baseline conditions, all models performed uniformly well. Under realistic narrative stress, performance diverged sharply, revealing two distinct stress-response phenotypes. Quantized models exhibited pseudonormalization, in which low flip rates hid functional collapse. Medical supervised fine-tuning systematically degraded logical stability, fairness, and information extraction. An open-weight model matched or exceeded proprietary alternatives on every safety dimension. These findings establish narrative stress auditing as a necessary complement to accuracy-based evaluation. Comments: 34 pages, 5 figures Subjects: Artificial Intelligence (cs.AI) ACM classes: I.2.7; J.3 Cite as: arXiv:2606.07929 [cs.AI]   (or arXiv:2606.07929v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.07929 Focus to learn more Submission history From: Linghua Yu Prof. [view email] [v1] Sat, 6 Jun 2026 01:39:14 UTC (3,384 KB) Access Paper: view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)