Beyond Surface Judgments: Human-Grounded Risk Evaluation of LLM-Generated Disinformation

Computer Science > Artificial Intelligence [Submitted on 8 Apr 2026] Beyond Surface Judgments: Human-Grounded Risk Evaluation of LLM-Generated Disinformation Zonghuan Xu, Xiang Zheng, Yutao Wu, Xingjun Ma Large language models (LLMs) can generate persuasive narratives at scale, raising concerns about their potential use in disinformation campaigns. Assessing this risk ultimately requires understanding how readers receive such content. In practice, however, LLM judges are increasingly used as a low-cost substitute for direct human evaluation, even though whether they faithfully track reader responses remains unclear. We recast evaluation in this setting as a proxy-validity problem and audit LLM judges against human reader responses. Using 290 aligned articles, 2,043 paired human ratings, and outputs from eight frontier judges, we examine judge--human alignment in terms of overall scoring, item-level ordering, and signal dependence. We find persistent judge--human gaps throughout. Relative to humans, judges are typically harsher, recover item-level human rankings only weakly, and rely on different textual signals, placing more weight on logical rigour while penalizing emotional intensity more strongly. At the same time, judges agree far more with one another than with human readers. These results suggest that LLM judges form a coherent evaluative group that is much more aligned internally than it is with human readers, indicating that internal agreement is not evidence of validity as a proxy for reader response. Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2604.06820 [cs.AI]   (or arXiv:2604.06820v1 [cs.AI] for this version)   https://doi.org/10.48550/arXiv.2604.06820 Focus to learn more Submission history From: Zonghuan Xu [view email] [v1] Wed, 8 Apr 2026 08:37:37 UTC (941 KB) Access Paper: HTML (experimental) view license Current browse context: cs.AI < prev   |   next > new | recent | 2026-04 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)