CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

Computer Science > Cryptography and Security [Submitted on 13 Mar 2026] CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities Arjun Chakraborty, Sandra Ho, Adam Cook, Manuel Meléndez CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33\% of the performance gap between smaller and larger models. Comments: 11 pages, 5 figures, 4 tables Subjects: Cryptography and Security (cs.CR) Cite as: arXiv:2603.13517 [cs.CR]   (or arXiv:2603.13517v1 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2603.13517 Focus to learn more Submission history From: Arjun Chakraborty [view email] [v1] Fri, 13 Mar 2026 18:48:40 UTC (84 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2026-03 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)