4 keys to integrating AI adoption risks into incident response planning - SC Media

COMMENTARY: It took only six months after the emergence of ChatGPT for a new class of incidents to emerge. The first well-publicized incident occurred in May 2023, when Samsung employees accidentally leaked confidential information by using ChatGPT to review internal code and documents. As a result, Samsung banned the use of generative AI (genAI) tools across the company to prevent future breaches. Numerous other events followed soon thereafter, each more damaging as the power of AI grows exponentially. In December 2023, a Chevrolet dealership’s AI chatbot was tricked into offering a $76,000 Tahoe for just $1. In February 2024, an Air Canada customer reportedly manipulated the company’s AI chatbot to obtain a refund larger than expected. More recently, an AI coding agent called Replit deleted an entire production database of business contacts without permission. [SC Media Perspectives columns are written by a trusted community of SC Media cybersecurity subject matter experts. Read more Perspectives here.] The rapid adoption of AI technologies is revolutionizing industries, offering unprecedented opportunities for innovation and efficiency. However, it also introduces unique risks that must be meticulously integrated into incident response planning, preparation, and post-event execution. For a comprehensive understanding of the various AI adoption risks, MIT published the AI Risk Repository late last year, which categorizes 700 such risks by cause and domain. As companies increasingly rely on AI tools, it is crucial to address the specific threats and challenges posed by these technologies to ensure robust and effective incident management. To navigate this new-look threat landscape, organizations must first understand the distinct nature of AI-related risks and adapt their incident response plans accordingly. The following outlines four key considerations for effectively managing these challenges across the incident response lifecycle. Understanding AI-specific risks As AI deployments move beyond pilot stages to general production, they introduce multi-dimensional risks. Generative AI models, such as Model Context Protocol (MCP) servers, which standardize AI context sharing across applications and platforms, increase the risk of prompt injection, privilege abuse, and token theft. Deepfakes and an increasing share of social engineering attacks are becoming more believable and more complex for users to distinguish truth from reality.  Additionally, adversaries can now exploit agentic AI to overwhelm security teams at a scale many believe will soon exceed the capacity of human-led SOCs. In response, some organizations may rush to deploy AI agents within the SOC. However, given the current stage of the technology’s maturity, it remains critical to keep humans in the loop. If an AI agent makes a mistake, the company bears the consequences. In contrast, adversaries face little to no repercussions when their agents fail; an unsuccessful attack simply means they try again. The use of AI tools, whether vendor-provided or internally developed, necessitates a comprehensive governance framework that includes AI-specific incident response components. These include continuous tracking of AI outputs for anomalies, bias or harmful behavior; clear protocols for human-in-the-loop escalation; AI-aware privacy protocols for handling data leaks; detailed logs of model inputs, outputs and decisions for forensic analysis; and more. In addition, companies must conduct thorough vendor due diligence, asking AI-specific questions and ensuring their incident response plans account for potential AI-related incidents. Incident response planning Effective incident response planning requires a thorough understanding of where AI is utilized across the business and the potential sources of AI-related incidents. This involves data mapping and identifying authorized AI usage, as well as addressing the challenges posed by shadow AI — unauthorized AI tools used by employees.  Companies must also consider the broader implications of AI incidents. These include ethical and societal impacts resulting from bias and discrimination; AI misuse that undermines human decision-making; liability for harm caused by autonomous AI agents; adversarial exploitation of AI model vulnerabilities; and violations of AI-specific laws, such as the EU’s AI Act or data privacy laws, including the GDPR. Preparation and training Preparation and training are critical for responding to AI incidents. This involves understanding the unique risks associated with the AI tools and models being used, whether they are vendor-provided or developed in-house.  AI is being integrated into various aspects of business operations, which means that an incident can simultaneously affect multiple departments. A well-coordinated response plan should involve all relevant teams, including those not typically part of incident response activities.  Given the distinct nature of AI incidents, provide specialized training to all team members who might be involved in a response, even those without prior experience in incident response. Education should encompass the specific challenges and nuances of AI incidents, including bias, discrimination, and compliance issues.  Finally, establish clear policies regarding AI use and implement robust monitoring systems to detect and manage unauthorized AI usage. This provides greater control over AI tools and ensures that employees are informed and confident about the AI technologies they use. Post-event execution AI-specific incidents often require organizations to engage a broader range of business teams than traditional incidents, necessitating enhanced cross-functional coordination during post-event response. This involves training and onboarding new stakeholders who may be unfamiliar with incident protocols, confidentiality, and the expectations for rapid response. Given AI’s unique governance and the enterprise-wide adoption push, organizations must ensure these diverse teams are prepared to participate effectively in incident mitigation, decision-making, and ongoing improvement, addressing challenges that are distinct to AI’s pervasive impact and evolving risk landscape. Balancing AI innovation and protection Integrating AI-specific risks into incident response planning, preparation, and post-event execution is imperative for comprehensive and effective incident management. As AI continues to evolve, companies must remain vigilant and proactive in addressing the unique challenges that these technologies pose. By doing so, they can harness the full potential of AI while safeguarding their operations and reputations.