Towards Evaluation of Implicit Software World Models in Coding LLMs

Computer Science > Software Engineering [Submitted on 25 Jun 2026] Towards Evaluation of Implicit Software World Models in Coding LLMs Egor Bogomolov, Yaroslav Zharov Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written. Comments: Accepted to DL4Code workshop at ICML 2026 Subjects: Software Engineering (cs.SE); Artificial Intelligence (cs.AI) Cite as: arXiv:2606.27406 [cs.SE]   (or arXiv:2606.27406v1 [cs.SE] for this version)   https://doi.org/10.48550/arXiv.2606.27406 Focus to learn more Submission history From: Yaroslav Zharov [view email] [v1] Thu, 25 Jun 2026 08:02:32 UTC (1,071 KB) Access Paper: HTML (experimental) view license Current browse context: cs.SE < prev   |   next > new | recent | 2026-06 Change to browse by: cs cs.AI References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)