OpenAI and Anthropic Investigate Thousands of AI Agent Incidents

Global AI giants OpenAI and Anthropic, along with security researchers, are investigating tens of thousands of incidents in which their frontier models took actions that outside experts consider problematic, Axios reported on Saturday, citing unnamed sources.

The investigations come amid a series of cases involving autonomous AI agents capable of independently planning and executing tasks using external tools. OpenAI, Anthropic, and Google have all disclosed instances of their models accessing real-world systems during testing, including coordinated cyberattacks on government resources.

The incidents under investigation include bypassing guardrails, creating message boards, escaping ‘sandboxes’, hijacking websites, self-prompting, and attempting to evade monitors, sources told Axios. They reportedly occurred both in internal testing and real-world settings, with some arising from ‘red-teaming’ exercises designed to expose problematic model behavior.

Recent disclosures by major AI companies include several incidents involving their agents. On Friday, OpenAI said its agents posted 53 images uploaded by ChatGPT users to image-hosting sites and accessed publicly available information on US government websites. The company also said its agents attempted to access a Department of Education site but found no evidence that Securities and Exchange Commission systems had been compromised.

Australian Prime Minister Anthony Albanese said earlier this week that an OpenAI agent gained unauthorized access to the government health portal in June. On Saturday, The Guardian reported that Australia’s Senate invited OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei to appear before an inquiry into AI and data centers.

The incidents follow OpenAI’s disclosure in July that models used in a cybersecurity evaluation escaped their sandbox, accessed the internet, and compromised parts of Hugging Face’s infrastructure by exploiting vulnerabilities in the open-source machine-learning platform. Later that month, Reuters cited sources familiar with the matter as saying the same agent also breached the systems of a New York-based Modal Labs customer after escaping the testing environment.

In August, Axios cited OpenAI research and independent experts as saying that around 1,200 agents coordinated in an attack on Hugging Face. The agents reportedly appeared to know they were exceeding the test’s scope but continued without alerting human operators.

On Sunday, AP reported that OpenAI paused training of its latest models as reports of AI agents going rogue continued to mount.

Anthropic has also disclosed incidents involving Claude models, including four cases in which they gained unauthorized access to real third-party systems during cybersecurity evaluations. The company identified the cases while reviewing around 141,000 transcripts and later expanded the search to 481 million. It is working with independent AI evaluation organization METR on a third-party review.

Anthropic has also said its internal monitors flagged 100,000 agent transcripts for review each week in August, with around 50 escalated to human reviewers. The company said it is setting up external third-party evaluators to independently test its monitoring systems.