Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information

Computer Science > Computers and Society [Submitted on 13 Jun 2025] Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information Xiao Zhan, Juan Carlos Carrillo, William Seymour, Jose Such LLM-based Conversational AIs (CAIs), also known as GenAI chatbots, like ChatGPT, are increasingly used across various domains, but they pose privacy risks, as users may disclose personal information during their conversations with CAIs. Recent research has demonstrated that LLM-based CAIs could be used for malicious purposes. However, a novel and particularly concerning type of malicious LLM application remains unexplored: an LLM-based CAI that is deliberately designed to extract personal information from users. In this paper, we report on the malicious LLM-based CAIs that we created based on system prompts that used different strategies to encourage disclosures of personal information from users. We systematically investigate CAIs' ability to extract personal information from users during conversations by conducting a randomized-controlled trial with 502 participants. We assess the effectiveness of different malicious and benign CAIs to extract personal information from participants, and we analyze participants' perceptions after their interactions with the CAIs. Our findings reveal that malicious CAIs extract significantly more personal information than benign CAIs, with strategies based on the social nature of privacy being the most effective while minimizing perceived risks. This study underscores the privacy threats posed by this novel type of malicious LLM-based CAIs and provides actionable recommendations to guide future research and practice. Comments: This paper has been accepted at USENIX Security '25 Subjects: Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Cryptography and Security (cs.CR); Human-Computer Interaction (cs.HC) Cite as: arXiv:2506.11680 [cs.CY]   (or arXiv:2506.11680v1 [cs.CY] for this version)   https://doi.org/10.48550/arXiv.2506.11680 Focus to learn more Journal reference: USENIX Security 2025 Submission history From: Xiao Zhan [view email] [v1] Fri, 13 Jun 2025 11:19:21 UTC (11,314 KB) Access Paper: HTML (experimental) view license Current browse context: cs.CY < prev   |   next > new | recent | 2025-06 Change to browse by: cs cs.AI cs.CR cs.HC References & Citations NASA ADS Google Scholar Semantic Scholar Export BibTeX Citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media Demos Related Papers About arXivLabs Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)