Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts

Computer Science > Artificial Intelligence [Submitted on 17 Feb 2026] Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts Eduardo Sardenberg, Antonio José Grandson Busson, Daniel de Sousa Moraes, Sérgio Colcher Smart contracts play a central role in blockchain systems by encoding financial and operational logic. Still, their susceptibility to subtle security flaws poses significant risks of financial loss and erosion of trust. LLMs create new opportunities for automating vulnerability detection, yet the effectiveness of different prompting strategies and model choices in real-world contexts remains uncertain. This paper evaluates state-of-the-art LLMs on Solidity smart contract analysis using a balanced dataset of 400 contracts under two tasks: (i) Error Detection, where the model performs binary classification to decide whether a contract is vulnerable, and (ii) Error Classification, where the model must assign the predicted issue to a specific vulnerability category. Models are evaluated using zero-shot prompting strategies, including zero-shot, zero-shot Chain-of-Thought (CoT), and zero-shot Tree-of-Thought (ToT). In the Error Detection task, CoT and ToT substantially increase recall (often approaching \approx 95--99\%), but typically reduce precision, indicating a more sensitive decision regime with more false positives. In the Error Classification task, Claude 3 Opus attains the best Weighted F1-score (90.8) under the ToT prompt, followed closely by its CoT. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2603.13239 [cs.AI]   (or arXiv:2603.13239v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2603.13239 Focus to learn more Submission history From: Antonio Busson [view email] [v1] Tue, 17 Feb 2026 18:08:56 UTC (149 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-03 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)