Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations

Computer Science > Artificial Intelligence [Submitted on 20 Apr 2026] Beyond One Output: Visualizing and Comparing Distributions of Language Model Generations Emily Reif, Claire Yang, Jared Hwang, Deniz Nazar, Noah Smith, Jeff Heer Users typically interact with and evaluate language models via single outputs, but each output is just one sample from a broad distribution of possible completions. This interaction hides distributional structure such as modes, uncommon edge cases, and sensitivity to small prompt changes, leading users to over-generalize from anecdotes when iterating on prompts for open-ended tasks. Informed by a formative study with researchers who use LMs (n=13) examining when stochasticity matters in practice, how they reason about distributions over language, and where current workflows break down, we introduce GROVE. GROVE is an interactive visualization that represents multiple LM generations as overlapping paths through a text graph, revealing shared structure, branching points, and clusters while preserving access to raw outputs. We evaluate across three crowdsourced user studies (N=47, 44, and 40 participants) targeting complementary distributional tasks. Our results support a hybrid workflow: graph summaries improve structural judgments such as assessing diversity, while direct output inspection remains stronger for detail-oriented questions. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2604.18724 [cs.AI]   (or arXiv:2604.18724v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2604.18724 Focus to learn more Submission history From: Emily Reif [view email] [v1] Mon, 20 Apr 2026 18:22:31 UTC (14,131 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-04 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)