Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning

Computer Science > Cryptography and Security [Submitted on 13 Jun 2026] Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuning Yiwei Chen, Lichi Li, Kai Cheung, Vinny Parla, Ganesh Sundaram We study the task of CVE-conditioned exploit generation, where a model drafts proof-of-concept (PoC) exploits given software vulnerability context. We adopt a data-centric approach, constructing a high-quality dataset via multi-stage preprocessing and introducing a scalable evaluation framework with LLM-as-judge and fine-grained rubrics. Under this unified setup, we benchmark 17 large language models across 8 evaluation criteria, providing systematic insights into their zero-shot capabilities. We further show that a compact 8B open-weight model, when fine-tuned on curated data, achieves over 42.5% improvement in exploit quality and rivals some proprietary models when combined with simple test-time rejection strategies. Our results highlight the importance of data quality, structured supervision, and evaluation design for reliable exploit generation, suggesting that these factors can be as critical as model scale in adapting LLMs to cybersecurity tasks. Comments: Technical Report Subjects: Cryptography and Security (cs.CR); Machine Learning (cs.LG) Cite as: arXiv:2606.15123 [cs.CR]   (or arXiv:2606.15123v1 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2606.15123 Focus to learn more Submission history From: Yiwei Chen [view email] [v1] Sat, 13 Jun 2026 05:28:20 UTC (1,178 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2026-06 Change to browse by: cs cs.LG References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)