Mitigating Many-Shot Jailbreaking

Computer Science > Machine Learning [Submitted on 13 Apr 2025 (v1), last revised 16 May 2025 (this version, v3)] Mitigating Many-Shot Jailbreaking Christopher M. Ackerman, Nina Panickssery Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a "fake" assistant responding inappropriately before the final request. With enough examples, the model's in-context learning abilities override its safety training, and it responds as if it were the "fake" assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training. Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR) Cite as: arXiv:2504.09604 [cs.LG]   (or arXiv:2504.09604v3 [cs.LG] for this version)   https://doi.org/10.48550/arXiv.2504.09604 Focus to learn more Submission history From: Christopher Ackerman [view email] [v1] Sun, 13 Apr 2025 14:42:03 UTC (1,736 KB) [v2] Wed, 7 May 2025 03:39:30 UTC (2,048 KB) [v3] Fri, 16 May 2025 01:12:44 UTC (1,635 KB) Access Paper: HTML (experimental) view license Current browse context: cs.LG < prev   |   next > new | recent | 2025-04 Change to browse by: cs cs.AI cs.CR References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)