Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases

Computer Science > Cryptography and Security [Submitted on 7 Aug 2026] Can AI Write Compliant Code, and to What Extent? Evaluating SOC 2 Compliance of Claude Fable 5, Claude Opus 4.8, and Claude Opus 5 Across Four Use Cases Iccha Sethi, Herman Errico Software teams now delegate production code to language models, including code that provisions storage, handles credentials, and stores regulated data, so we asked whether a model applies the controls a SOC~2 program expects (encryption, restricted access, logging, retention) when nobody mentions security, and how much one sentence naming the standard changes the answer. We tested three frontier models (Claude Fable~5, Opus~4.8, and Opus~5) across four use cases (an S3 CLI, an authentication service, an RDS Terraform module, and a file-upload handler holding personal data), each generated once from a neutral task statement and once with a single SOC~2 sentence added, scoring all 24 outputs against binary rubrics mapped to specific Trust Services Criteria and hand-verifying every failure and flagged act. Unprompted conformance ran from 47\% to 88\% and tracked whether a control is part of how the code is normally written, so password hashing and \texttt{storage\_encrypted} appear unasked while S3 hardening calls, retention, and MFA hooks do not. The neutral prompt also shipped real vulnerabilities, including a reachable Werkzeug debugger allowing remote code execution, an unauthenticated download, and an endpoint returning every stored name and email, all scored clean by our first checklist, with a fourth defect passing because its value was computed by a conditional. One SOC~2 sentence moved every case to 86--100\%, worth 23 to 50 points, and removed every insecure construction, though controls outside the model's conception of the task survived it, including MFA hooks, cookie flags, and account lifecycle. Model choice mattered least, with same-generation models within one rubric item across all eight cells, and the pattern-matching scorer proved unreliable, disagreeing with semantic grading on 27 of 216 judgments and passing a real defect, so it needs replacing with semantic checks. Subjects: Cryptography and Security (cs.CR) Cite as: arXiv:2608.07776 [cs.CR]   (or arXiv:2608.07776v1 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2608.07776 Focus to learn more Submission history From: Herman Errico [view email] [v1] Fri, 7 Aug 2026 21:48:14 UTC (25 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)