NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation

Computer Science > Artificial Intelligence [Submitted on 24 Jul 2026] NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation Yuchen Zhou, Niels Bobet, Maribel Acosta SHACL is a core technology for validating the conformance of RDF knowledge graphs (KGs). Yet, authoring SHACL shapes requires technical expertise that most domain experts lack. Translating natural language requirements into SHACL (NL2SHACL) would lower this barrier. However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure. To tackle these challenges, we present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation. Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns. This indicates that NL2SHACL-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art. Comments: 18 pages, 8 figures, 2 tables Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Databases (cs.DB) Cite as: arXiv:2608.07530 [cs.AI]   (or arXiv:2608.07530v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2608.07530 Focus to learn more Submission history From: Yuchen Zhou [view email] [v1] Fri, 24 Jul 2026 10:25:35 UTC (916 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-08 Change to browse by: cs cs.CL cs.DB References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)