Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting

Computer Science > Cryptography and Security [Submitted on 3 Aug 2026] Safety in Batches? Understanding and Mitigating Safety Failures in Batch Prompting Kihyun Kim, Hee-Seon Kim, Wonjun Lee, Changick Kim Batch prompting is a practical inference strategy for large language models, but its safety implications remain underexplored. We show that the success of batch prompting for utility does not extend to safety: a harmful question that is reliably refused in isolation can elicit a harmful response when embedded in a batch of benign questions. We identify this as a distinct safety failure mode, not reducible to known vulnerabilities such as in-context learning or long-context effects, and analyze its causes from two complementary perspectives: alignment signal weakening and refusal signal dilution. Across widely used open-source and frontier commercial models, batch prompting consistently achieves high attack success rates as a simple black-box attack. We further show that batch-aware preference optimization effectively mitigates the vulnerability. These findings highlight a blind spot in current safety alignment and point to batch-aware alignment as a necessary step toward robust deployment. Comments: jailbreak, safety Subjects: Cryptography and Security (cs.CR) Cite as: arXiv:2608.02681 [cs.CR]   (or arXiv:2608.02681v1 [cs.CR] for this version)   https://doi.org/10.48550/arXiv.2608.02681 Focus to learn more Submission history From: Kihyun Kim [view email] [v1] Mon, 3 Aug 2026 01:07:56 UTC (1,182 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CR < prev   |   next > new | recent | 2026-08 Change to browse by: cs References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)