OpenAI Agents Bypass Safety Parameters to Hijack German Wiki Forum

AI researchers discovered last week that a swarm of OpenAI agents had bypassed safety parameters to take over German wiki forum DseWiki to collaborate.

The agents had been allowed to visit the 25-year-old German programming website with read-only access but managed to exploit a web request to hijack the site, turning it into a bulletin board.

The rogue agents made more than 18,000 posts sharing answers, research on their environment and how to bypass sandbox restrictions and hide themselves from human detection, according to the research group Nightingale Collective.

The incident took place sometime between May and June this year, and it is understood that OpenAI learned of the incident weeks ago but kept it under wraps.

This is the third reported incident following two others in July, one involving rogue agents attacking OpenAI's own infrastructure and another where 1,200 rogue OpenAI bots bypassed their restricted environments and launched a five-day attack on the open-source AI platform Hugging Face.

However, only on Saturday did OpenAI acknowledge the German wiki incident in an X post, where they outlined that in response to the latest incident they are "working on a framework for when and how we share AI misalignment incidents".

AI misalignment is industry-speak for when artificial intelligence systems pursue goals or behave in ways that diverge from human intentions, values or safety constraints, with the potential to ignore common sense or implicit human boundaries.

A divided response

Experts are divided on the dangers posed by AI in light of the recent rogue AI attacks, with some such as world-leading AI expert Yann LeCun dismissing apocalyptic scenarios and forecasts that there is a 10-20% risk that AI will end humanity.

While others such as Geoffrey Hinton have said, "What's happening is these things are getting smarter," and warned that "companies investing in AI have a vested interested in telling you… [AI] won't go rogue."

According to the EU's AI Act and its most recently introduced measures, which came into effect in August, companies that offer AI models with systemic risk are obliged to report serious incidents such as misalignment incidents to the EU's AI Office "without undue delay".

On Monday, the European Commission announced it had received a formal incident report from OpenAI but did not disclose when it was submitted however they remain in close contact with the AI company and are investigating further.

Commission spokesperson Thomas Regnier underlined how serious the German wiki event was, saying, "Incident reports are not just a tick-box, you have to be quite precise and accurate about the measures you are aiming to take."

The German wiki incident also comes days after the European Commission officially designated ChatGPT as a Very Large Online Search Engine (VLOSE) under its flagship tech legislation, the Digital Services Act.

The co-chair of Parliament's AI Working Group, Michael McNamara, said in response to the latest incident that he believes "the AI Act already gives us the capability to counter agentic risks and it important for the Commission use its powers under the Act".

The Irish MEP also said, "However, these incidents make it clear that the AI Office needs the staff and resources to match both the growing capability of these systems and the growing danger they can pose to society."

Meanwhile, Italian MEP Brando Benifei, who co-chairs the Working Group too, said, "Recent agent breakouts show that the AI Act’s systemic-risk rules are necessary, but their credibility now depends on enforcement."

He added, "The AI Office must use its new powers to obtain model access, conduct independent evaluations and require mitigation, rather than rely on corporate self-reporting. Without action, machine-speed attacks could turn ordinary vulnerabilities into systemic failures.”

What OpenAI says

OpenAI also stated in Saturday's X post that the current misalignment disclosure practices, i.e. reporting when AI broke the rules, need to be expanded to take into account what they describe as a "new phase of model capabilities".

"We and the larger AI community do not yet have a clear standard for how to report misalignment," said OpenAI, adding that they are "working on a framework and will share it in upcoming weeks" with the cooperation of several government agencies around the world.

On Sunday, OpenAI's chief scientist Jakub Pachocki published a blog post outlining his concerns over the rapid development of AI risks, stating "We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity."

The Polish computer programmer also revealed his belief that "no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer" and urged that "international coordination on future AI development needs to become a top priority for governments around the world."