Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

Computer Science > Cryptography and Security [Submitted on 10 Jun 2026] Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment Derek Yohn, Luke Flancher, Mirajul Islam, Khaled Slhoub This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions. Comments: Keywords: Agentic AI, Cybersecurity, Large Language Models, Static Application Security Testing, Model performance evaluation Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) Cite as: arXiv:2606.11672 [cs.CR]   (or arXiv:2606.11672v1 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2606.11672 Focus to learn more Submission history From: Khaled Slhoub Dr [view email] [v1] Wed, 10 Jun 2026 05:31:24 UTC (117 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2026-06 Change to browse by: cs cs.AI References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)