Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration

Computer Science > Artificial Intelligence [Submitted on 8 Mar 2026] Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration Bakhtawar Ahtisham, Kirk Vanacore, Rene F. Kizilcec Large language models (LLMs) are increasingly positioned as scalable tools for annotating educational data, including classroom discourse, interaction logs, and qualitative learning artifacts. Their ability to rapidly summarize instructional interactions and assign rubric-aligned labels has fueled optimism about reducing the cost and time associated with expert human annotation. However, growing evidence suggests that single-pass LLM outputs remain unreliable for high-stakes educational constructs that require contextual, pedagogical, or normative judgment, such as instructional intent or discourse moves. This tension between scale and validity sits at the core of contemporary education data science. In this work, we present and empirically evaluate a hierarchical, cost-aware orchestration framework for LLM-based annotation that improves reliability while explicitly modeling computational tradeoffs. Rather than treating annotation as a one-shot prediction problem, we conceptualize it as a multi-stage epistemic process comprising (1) an unverified single-pass annotation stage, in which models independently assign labels based on the rubric; (2) a self-verification stage, in which each model audits its own output against rubric definitions and revises its label if inconsistencies are detected; and (3) a disagreement-centric adjudication stage, in which an independent adjudicator model examines the verified labels and justifications and determines a final label in accordance with the rubric. This structure mirrors established human annotation workflows in educational research, where initial coding is followed by self-checking and expert resolution of disagreements. Comments: Accepted for presentation at the Education Data Science Conference (EDS 2026), Stanford, USA, May 26-28, 2026. Extended abstract Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2603.13353 [cs.AI]   (or arXiv:2603.13353v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2603.13353 Focus to learn more Submission history From: Bakhtawar Ahtisham [view email] [v1] Sun, 8 Mar 2026 16:51:03 UTC (857 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-03 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)