Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models

Computer Science > Artificial Intelligence [Submitted on 17 Apr 2026] Characterising LLM-Generated Competency Questions: a Cross-Domain Empirical Study using Open and Closed Models Reham Alharbi, Valentina Tamma, Terry R. Payne, Jacopo de Berardinis Competency Questions (CQs) are a cornerstone of requirement elicitation in ontology engineering. CQs represent requirements as a set of natural language questions that an ontology should satisfy; they are traditionally modelled by ontology engineers together with domain experts as part of a human-centred, manual elicitation process. The use of Generative AI automates CQ creation at scale, therefore democratising the process of generation, widening stakeholder engagement, and ultimately broadening access to ontology engineering. However, given the large and heterogeneous landscape of LLMs, varying in dimensions such as parameter scale, task and domain specialisation, and accessibility, it is crucial to characterise and understand the intrinsic, observable properties of the CQs they produce (e.g., readability, structural complexity) through a systematic, cross-domain analysis. This paper introduces a set of quantitative measures for the systematic comparison of CQs across multiple dimensions. Using CQs generated from well defined use cases and scenarios, we identify their salient properties, including readability, relevance with respect to the input text and structural complexity of the generated questions. We conduct our experiments over a set of use cases and requirements using a range of LLMs, including both open (KimiK2-1T, LLama3.1-8B, LLama3.2-3B) and closed models (Gemini 2.5 Pro, GPT 4.1). Our analysis demonstrates that LLM performance reflects distinct generation profiles shaped by the use case. Comments: arXiv admin note: text overlap with arXiv:2507.02989 Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2604.16258 [cs.AI]   (or arXiv:2604.16258v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2604.16258 Focus to learn more Submission history From: Reham Alharbi Miss [view email] [v1] Fri, 17 Apr 2026 17:16:32 UTC (65 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-04 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)