DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain

Computer Science > Cryptography and Security [Submitted on 18 Apr 2025 (v1), last revised 26 Jun 2026 (this version, v4)] DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain Enhao Huang, Pengyu Sun, Shuxun Wang, Zixin Lin, Alex Chen, Kaichun Hu, Joey Ouyang, Frank Li, Zhiyu Zhang, Haobo Wang, Yiming Li, Zhan Qin, James Yi, Gang Zhao, Ziang Ling, Lowes Yang The Web3 ecosystem, underpinned by cryptographic primitives and decentralized consensus, represents a high-stakes environment where software vulnerabilities and incentive misalignments translate directly into financial loss. As Large Language Models (LLMs) are increasingly integrated into this domain for tasks ranging from smart contract auditing to decentralized finance analytics, ensuring their reliability is paramount. However, general-purpose benchmarks fail to capture the specialized reasoning required for these adversarial and protocol-driven settings. To bridge this gap, we introduce DMind Benchmark, a comprehensive evaluation suite designed to rigorously assess LLM proficiency across the Web3 stack. DMind Benchmark encompasses nine distinct subdomains (spanning infrastructure, smart contracts, token economics, etc.) and combines objective knowledge retrieval with complex open-ended reasoning tasks that emulate real-world operational challenges. We conduct an extensive evaluation of 31 leading proprietary and open-weights models, employing a contamination-aware pipeline and verifying the statistical robustness of our scoring protocol through rigorous cross-judge consistency checks. Our analysis reveals a critical dichotomy: while models demonstrate competence in foundational infrastructure concepts, they exhibit significant vulnerabilities in high-reasoning tasks such as security auditing. Furthermore, we provide a Pareto analysis to guide cost-effective deployment and demonstrate through adversarial experiments that high performance on DMind Benchmark necessitates genuine reasoning rather than superficial memorization. Since its open-source release in April 2025, DMind Benchmark achieved the #1 trending position on Hugging Face for nearly a week and accumulated over 13k downloads by June 2026, establishing itself as a standard for advancing secure and trustworthy AI in Web3. Subjects: Cryptography and Security (cs.CR); Artificial Intelligence (cs.AI) Cite as: arXiv:2504.16116 [cs.CR]   (or arXiv:2504.16116v4 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2504.16116 Focus to learn more Submission history From: Yihui Yang [view email] [v1] Fri, 18 Apr 2025 16:40:39 UTC (1,541 KB) [v2] Fri, 16 May 2025 12:00:59 UTC (1,085 KB) [v3] Tue, 4 Nov 2025 16:26:20 UTC (5,861 KB) [v4] Fri, 26 Jun 2026 10:59:58 UTC (17,655 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2025-04 Change to browse by: cs cs.AI References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)