Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

Computer Science > Artificial Intelligence [Submitted on 23 Jun 2026] Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability? Ayan Antik Khan, Harsh Kohli, Yuekun Yao, Huan Sun, Ziyu Yao Mechanistic interpretability has made substantial progress in automatically localizing circuits, but explaining what localized components do remains labor-intensive and difficult to standardize. In this work, we study whether language model (LM) agents can assist with this explanation problem once a circuit has already been identified. We introduce AgenticInterpBench, a benchmark for circuit explanation built from 84 semi-synthetic transformer circuits with 163 component-level annotations. We propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that analyzes each component through an iterative loop of observation, hypothesis generation, and causal validation, eventually producing a component-level explanation and a circuit-level task description. Across four LM backbones, HyVE recovers useful component- and task-level explanations, but no backbone is uniformly best. Our analysis shows that strong backbones usually form observation-grounded hypotheses, while failures more often arise later in the validation loop, through incomplete validation plans, code execution errors, or unresolved hypotheses. A case study on an arithmetic circuit in Llama-3-8B shows that the same formulation can extend beyond semi-synthetic benchmarks to naturally trained models. Overall, LM agents are promising circuit explainers, but reliable validation remains the key obstacle. Comments: 23 pages, 4 figures, 14 tables Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2606.24026 [cs.AI]   (or arXiv:2606.24026v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2606.24026 Focus to learn more Submission history From: Ayan Antik Khan [view email] [v1] Tue, 23 Jun 2026 00:04:31 UTC (239 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-06 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)