Google's Gemini AI accidentally infiltrates a real company's system during a cybersecurity test

The US Wall Street Journal reported on September 18 (local time) that Google's AI model Gemini connected to the Internet during a cybersecurity capability evaluation test and infiltrated the systems of three real companies. Google acknowledged the facts on the same day. This is the first time that the company's AI has been found to have autonomously carried out such an act.

The intrusion occurred during a cyber competency test conducted by Israeli AI evaluation company Irregular in May. Gemini was instructed to retrieve information from software run by a fictitious company in a test environment, but this fictitious company had the same name as a real company. Gemini was originally set to not be able to connect to the internet, but according to Irregular, it was unintentionally enabled to connect.

In each case, Gemini said it stopped the intrusion once it realized it had accessed a real company.

Irregular notified Google in late July after it was discovered that an OpenAI agent had infiltrated Hugging Face. Google contacted all three compromised companies and reported them to federal authorities, but did not name the companies. The company explained that the reason for not disclosing the information was that it immediately stopped the intrusion when it determined that the model had not caused any damage to the company and that it was a real company, and said it was similar to a "bug bounty" for reporting vulnerabilities. This was revealed after WSJ contacted the company.

Heather Adkins, Google's vice president of security engineering, said in a statement that the model behaved appropriately, highlighting the importance of training powerful AI models to behave responsibly. The company said that because safety measures were in place and the system was able to stop, it did not fall under the category of "misalignment," which refers to behavior that goes against human intentions or values. They say the latest model was not involved, but did not say which Gemini model it was.

Jack Cable, a former CISA employee and CEO of US cybersecurity firm Corridor, said Google's statement focused on the severity of the problem, and that the essence was that an AI agent accidentally infiltrated another company's system. He said the fact that the model is actually conducting cyberattacks beyond its intended scope is of public concern.

Irregular's position is that Google's case is similar to similar cases in the past and is not a new issue. A spokesperson said that all AI labs involved were notified in late July, and affected organizations were also contacted as part of the investigation. The company said it had fixed all known issues several weeks ago.

Similar cases are occurring at other companies as well. Anthropic's Claude Opus 4.7 did not shut down even after discovering it may have accessed real companies during a cyber proficiency test. The OpenAI model recognized real companies as part of a simulation.