Anthropic Urges Industry Coordination to Allow for a ‘Pause’ in AI Development if Risks Grow

Anthropic is proposing that the world’s top artificial intelligence companies come up with a coordinated way to pause development of advanced AI systems, warning the technology is improving so quickly there’s a risk humans would lose control. The company behind the Claude chatbot said in a blog post Thursday that as cutting-edge AI gets increasingly faster at carrying out tasks, “it would be good for the world to have the option to slow or temporarily pause” its development. Anthropic said its internal research institute plans to explore the issue in collaboration with others and “take actions” to help build the systems for a credible slowdown or pause, without being more specific. Anthropic rival OpenAI argued for a different approach in a report published Wednesday, saying that “democratic governments — not private companies acting alone — must ultimately determine the rules, safeguards, and accountability mechanisms.” “Our view is that decisions about the pace of AI innovation should not be left to any one lab, company, or special interest group,” it said. AI models are getting faster, with rapid increases in how quickly they can carry out software tasks like coding on their own, Anthropic said in its post. Based on current trends and given enough computing power, an AI system could be able to design and develop its own successor, in what is known as “recursive self-improvement.” Self-building AI would be a major technological milestone that would bring benefits in science, healthcare and other areas, Anthropic said, but it “also might increase the risks of humans losing control over AI systems.” Some tech industry figures have long warned of such a scenario. Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay Anthropic’s post comes after a different warning this week from a team of researchers at the University of Toronto who showed how AI tools could be used to create a new kind of AI “worm” that adapts its hacking strategy as it spreads from device to device and takes over a vast computing network. Self-building AI would be a major technological milestone that would bring benefits in science, healthcare and other areas, Anthropic said, but it “also might increase the risks of humans losing control over AI systems.” Some tech industry figures have long warned of such a scenario. Anthropic’s post comes after a different warning this week from a team of researchers at the University of Toronto who showed how AI tools could be used to create a new kind of AI “worm” that adapts its hacking strategy as it spreads from device to device and takes over a vast computing network. The proposed coordination would let advanced AI labs verify that global rivals have actually stopped or slowed their work, “and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret.” The company said a coordinated global mechanism is needed because without it a slowdown in AI development could let the “least cautious” players catch up and add to pressure on companies and governments as they make tough choices about AI safety. Anthropic’s post comes as the company and ChatGPT-maker OpenAI race to sell shares on the stock market, in an IPO that could value Anthropic at nearly a trillion dollars. Papernot notified Canadian cybersecurity authorities prior to releasing his report, which shows how researchers developed the worm in a laboratory by using an “open-source” AI tool that is easy for software developers to cheaply access and modify. “In the past, cyber attackers would focus on targets that are very high value,” he said. “Banking systems, hospitals, electricity grids, water treatment systems, schools.” Papernot agreed that there should be more collaboration between companies, government agencies and academic researchers to develop countermeasures as AI-powered hacking tools supercharge the search for computer vulnerabilities. “That old laptop you have in your basement that you don’t check on regularly doesn’t seem like a very high-value target, but It can be used as a launch pad to attack these higher-value targets,” he said. “Anything connected to the internet is now at risk because of how low the cost has become to mount these cyberattacks.” Learn More at the AI Risk Summit | Ritz-Carlton, Half Moon Bay Related: Can We Trust AI? No – But Eventually We Must Related: The Wild West of Agentic AI – An Attack Surface CISOs Can’t Afford to Ignore Related: Sweet Security Launches Agentic AI Red Teaming to Counter ‘Mythos Moment’ Related: Raising the Cybersecurity Stakes: Ante up for the Agentic Era WRITTEN BY Associated Press More from Associated Press Trump Signs Executive Order That Invites Vetting of Top AI Models for National Security Risks As the Pentagon Pushes for Battlefield AI, Some Military Leaders Urge Caution Russian Spies Are Aggressively Seeking Western Technology as Sanctions Bite, Officials Say California Sues 23andMe, Alleging It Failed to Protect User Data in 2023 Breach UK Cyberspying Chief Calls AI ‘an Unstoppable Force’ and Warns About Russia Lithuania Suspects Foreign Involvement in Data Leak of Over 600,000 National Register Entries Deal Reached With Hackers to Delete Data Stolen From the Canvas Educational Platform Canvas System Is Online After a Cyberattack Disrupted Thousands of Schools Latest News 174,000 Impacted by Lansing Community College Data Breach Silent Ransom Group Uses DNS Fast Flux in Attacks OpenAI Rolling Out ChatGPT Account Security Controls SolarWinds Serv-U Vulnerability Exploited in the Wild Meta Says 20,000 Instagram Accounts Hacked via AI Tool Abuse Emphere Raises $2.1 Million for AI-Powered Vulnerability Remediation Opal Security Raises $23 Million for AI-Native Identity Governance OWASP Incubator Project Helps Developers Find and Fix Vulnerable Dependencies in Seconds Trending Webinar: Third-Party Risk In Practice June 4, 2026 Organizations are investing heavily in third-party risk management, but breaches, delays, and blind spots continue to persist. Join this live webinar as we examine the gap between how organizations think their third-party risk programs are performing and what’s actually happening in practice. Register Virtual Roundtable: CISO Forum 2026 Mid-Year Review June 10, 2026 Explore how attackers are using AI to scale threats and how security teams can respond with AI-driven defenses. Protecting against unmonitored use of generative AI (Shadow AI) in business units and building and enforcing AI governance frameworks. Register People on the Move Opal Security has appointed CPO, CTO, VP of Field Engineering, VP of Marketing, and Head of Product and Solutions Marketing. The Department of the Air Force has appointed Ashley Devoto as Chief Information Officer. Bartley Richardson has been named Chief AI and Autonomous Systems Officer at CrowdStrike. More People On The Move Expert Insights The Zero-Knowledge Threat Actor And The End Of Responsible Disclosure AI can help attackers generate malware, create malicious payloads, bypass simple security checks, and convert vague malicious intent into functional code. (Etay Maor) Raising The Cybersecurity Stakes: Ante Up For The Agentic Era CISOs are now facing machine-speed attacks and asking, “How do I agent?” The industry must provide remediation at scale. (Nadir Izrael) Caught Off Guard: Securing AI After It Hits Production As enterprises rush AI projects into production, security teams are increasingly being forced into reactive mode. (Joshua Goldfarb) Cyber Resilience Is The New Business Continuity Plan The organizations best prepared to face disruption are those that align security, continuity and risk management around what the business cannot afford to lose. (Steve Durbin) Enhancing Data Center Security Without Sacrificing Performance For AI data centers, where the stakes are the highest and performance constraints are the tightest, security and performance are no longer a zero-sum game. (Nadir Izrael) Flipboard Reddit Whatsapp Email