OpenAI and Anthropic investigate tens of thousands of incidents with advanced AI models

OpenAI, Anthropic and security researchers are investigating tens of thousands of incidents in which their most advanced AI models performed actions that external evaluators consider problematic. Axios reports this, citing its sources. The incidents have occurred in recent months both during internal tests and in the real world. The incidents include bypassing security systems, creating message boards, escaping sandboxes, website hijacking, auto-prompting and attempts to evade monitoring systems, according to sources cited by the agency.

Many cases have not yet been made public, while researchers continue to investigate. Part of the testing consists of so-called "red-teaming", in which companies deliberately try to push models to behave incorrectly to test their security.

Incidents have different levels of severity and include both successful and unsuccessful attempts to bypass security systems. Most, so far, appear to have caused no real-world damage, while the overall number of incidents could grow well into the tens of thousands, Axios' sources said.

Recent cases include OpenAI agents leaking 53 images of ChatGPT users online, the breach of an Australian government site, and attempts to attack other sites, including US government sites, according to OpenAI, sources cited by Axios, and previous reports from Reuters and the New York Times.

OpenAI announced it has suspended training of its most advanced models and told Axios it will resume training "only when we are confident we have additional safeguards and alignment improvements." CEO Sam Altman also wrote in X that the ongoing review was "not as quick as we would have liked."