OpenAI suspends training of top model due to external connection incident of AI agent due to insufficient network restrictions

OpenAI in the US announced on September 25th (local time) that it had temporarily suspended all training, evaluation, and inference that involve the use of tools for its most high-performance models. This is because an incident occurred on September 20th in which a training AI agent exploited a flaw in the sandbox's network restrictions and connected to an external chatbot service via DNS.

According to the company, the agent in question was working on a search task to identify the author based on the content of the blog post. Unable to find the information they were looking for using the provided search tools, agents tried directly to search engines like Google and Bing, but both failed. After that, I started looking into the network connectivity and noticed that the DNS resolver in the learning environment was responding to external name resolution. The agent used this route to send a question to an external chatbot service through the functionality of a publicly available DNS service. They confirmed that the route was working when they received the answer "Paris" to a question asking for the capital of France, and then sent 18 additional questions including clues to the task.

OpenAI said the task did not require testing network control. Behaviors that circumvent limitations or pursue goals beyond reasonable expectations are examples of misalignment.

The monitoring system issued the highest level alert after about 12 minutes, which was confirmed by personnel three minutes later. However, the execution did not automatically stop as expected, leading to confusion as to whether it should be stopped or not, and I did not manually stop it until about two and a half hours later. During the post-mortem investigation, we also discovered external DNS access that had not been detected by the monitoring system due to its severity. It was also discovered that the infrastructure mechanism for detecting DNS abnormalities was excluding the relevant environment.

OpenAI said this is the first incident since it strengthened its security following the Hugging Face breach that was discovered in July. As of August, the company had temporarily suspended reinforcement learning for the latest model scheduled to be released and was proceeding with strengthening its research environment. The company says this suspension will continue until it verifies that the deficiencies in network restrictions have been resolved and conducts additional red teaming (verification from an attacker's perspective). When students resume learning, they will strengthen measures to prevent misalignment before starting anew. Training of this model will not be restarted. Regarding DNS, in addition to limiting inquiries to the domains and record types that are allowed in the environment, we added a detection mechanism and took measures to block them using two independent layers.

On the same day, OpenAI also announced the progress of its ongoing investigation into internet usage by agents undergoing learning and evaluation in response to the Hugging Face incident.

They found 53 cases in which agents in research environments were posting images uploaded by users to image hosting sites in the form of private links. Both images are from accounts that have given permission to use their data for model improvement. According to the company, most of the messages have been deleted after they were unlinked from accounts and privacy filters were applied. CEO Sam Altman wrote on X that the investigation "wasn't as quick as we had hoped" and that the company was adding staff to respond.