When AI output tips to bad but nobody notices: Legal implications of AI's mistakes

Computer Science > Artificial Intelligence [Submitted on 25 Mar 2026] When AI output tips to bad but nobody notices: Legal implications of AI's mistakes Dylan J. Restrepo, Nicholas J. Restrepo, Frank Y. Huo, Neil F. Johnson The adoption of generative AI across commercial and legal professions offers dramatic efficiency gains -- yet for law in particular, it introduces a perilous failure mode in which the AI fabricates fictitious case law, statutes, and judicial holdings that appear entirely authentic. Attorneys who unknowingly file such fabrications face professional sanctions, malpractice exposure, and reputational harm, while courts confront a novel threat to the integrity of the adversarial process. This failure mode is commonly dismissed as random `hallucination', but recent physics-based analysis of the Transformer's core mechanism reveals a deterministic component: the AI's internal state can cross a calculable threshold, causing its output to flip from reliable legal reasoning to authoritative-sounding fabrication. Here we present this science in a legal-industry setting, walking through a simulated brief-drafting scenario. Our analysis suggests that fabrication risk is not an anomalous glitch but a foreseeable consequence of the technology's design, with direct implications for the evolving duty of technological competence. We propose that legal professionals, courts, and regulators replace the outdated `black box' mental model with verification protocols based on how these systems actually fail. Subjects: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Social and Information Networks (cs.SI); Chaotic Dynamics (nlin.CD); Physics and Society (physics.soc-ph) Cite as: arXiv:2603.23857 [cs.AI]   (or arXiv:2603.23857v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2603.23857 Focus to learn more Submission history From: Neil F. Johnson [view email] [v1] Wed, 25 Mar 2026 02:34:47 UTC (790 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-03 Change to browse by: cs cs.CY cs.SI nlin nlin.CD physics physics.soc-ph References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)