Position: Science of AI Evaluation Requires Item-level Benchmark Data

Computer Science > Artificial Intelligence [Submitted on 27 Feb 2026] Position: Science of AI Evaluation Requires Item-level Benchmark Data Han Jiang, Susu Zhang, Xiaoyuan Yi, Xing Xie, Ziang Xiao AI evaluations have become the primary evidence for deploying generative AI systems across high-stakes domains. However, current evaluation paradigms often exhibit systemic validity failures. These issues, ranging from unjustified design choices to misaligned metrics, remain intractable without a principled framework for gathering validity evidence and conducting granular diagnostic analysis. In this position paper, we argue that item-level AI benchmark data is essential for establishing a rigorous science of AI evaluation. Item-level analysis enables fine-grained diagnostics and principled validation of benchmarks. We substantiate this position by dissecting current validity failures and revisiting evaluation paradigms across computer science and psychometrics. Through illustrative analyses of item properties and latent constructs, we demonstrate the unique insights afforded by item-level data. To catalyze community-wide adoption, we introduce OpenEval, a growing repository of item-level benchmark data designed supporting evidence-centered AI evaluation. Subjects: Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Databases (cs.DB) Cite as: arXiv:2604.03244 [cs.AI]   (or arXiv:2604.03244v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2604.03244 Focus to learn more Submission history From: Han Jiang [view email] [v1] Fri, 27 Feb 2026 04:31:30 UTC (1,356 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-04 Change to browse by: cs cs.CY cs.DB References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)