OpenAI's Rogue AI Agent and Mounting Corporate Challenges

Throughout history, many things have been seen by terrified populaces as a harbinger of doom. A comet. A crow on the battlefield. A solar eclipse. A mutant livestock birth. Yet times move on. In the modern era, the leading harbinger of doom is literally any picture of the OpenAI CEO, Sam Altman, attached to a news story. You know it’s not going to be good, right? You know that by the time you’ve read it, you’ll be begging to go back to the time when the worst thing that could happen to us at the hands of the techlords was just some democracy-subversion, or childhood destruction, usually followed by Mark Zuckerberg putting on a suit and claiming: “We will learn from this.”

Anyway: a lot of pictures of Sam Altman in the news of late. Most recently, this week, one darkened the skies alongside the tale of how an OpenAI autonomous agent went rogue during a supposedly sandboxed/guardrailed test, and hacked a major startup that functions as a repository of coding information. (I’m slightly obsessed with the fact that the startup in question is called Hugging Face, adding weight to my suspicion that some vast, tweely benign emoji is the last face humanity will see before it dies.)

Needless to say, an affectless OpenAI statement broke the news. “We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of,” this droned calmly, trundling on toward its inevitable “learnings”. “We’re improving and adding stronger protections around future training and evaluations.” It’s such a particular tone, isn’t it, dressed in the psychopathically desiccated language of management speak. Like having to listen to a homicidal sex criminal talk about killing people by “close of play”, and then “circling back” to victims to remove a trophy.

Then again, for a company that likes to present itself as the planet’s leading agent of revolutionary prosperity when things go right, OpenAI lapses tellingly into the passive when things go wrong. Bad things seem to happen to it, not because of it, at which point it takes on the role of tirelessly unflappable investigator, like one of those firemen who moonlights as a serial arsonist.

Amazingly, it was possible to detect even a note of self-congratulation in OpenAI’s take on the whole situation. “We consider this incident to be an unprecedented cyber-incident,” the company’s statement remarked, “involving state-of-the-art cyber capabilities.” I don’t know what you’d call this general vibe for Earthlings. Death by humblebrag? I increasingly feel we’ll find out the answer to the question “What’s the worst that can happen?” in a blog post from OpenAI entitled “OpenAI first to discover the worst that can happen”. Already, a significant section of AI watchers are convinced that the only reason OpenAI would tell the world about this incident is as a marketing tool, or even a come-and-get-me plea for regulation that will in effect protect them and burn the bridge behind them for smaller competitors.

After all, these are testing times for the firm. This is a month in which Open AI was discovered to be exploiting a legal loophole to sell its advanced AI models to Chinese tech firms blacklisted by the Pentagon. S&P Global Ratings cited OpenAI as a “key credit risk” in the course of downgrading US tech giant Oracle to BBB-, which is one notch above junk status. And it seems to be on track to miss its five-year ad revenue projection by 90 – NINETY – per cent. Which, without getting too financially technical, feels like a lot of per cent. And it’s not great news on the hardware front, either, with Apple suing it, alleging that its consumer hardware plans are based on stolen intellectual property. Meanwhile China’s much cheaper provider, DeepSeek, is believed to be preparing for an IPO, maybe even filing this year.

Back to the old worst-that-can-happen question again, though, with a reminder that the Pentagon dramatically binned off and threatened to destroy another AI firm, Anthropic, earlier this year after it resisted loosening its ethical guidelines that prevented use of its technology for, among other things, autonomous lethal weapons. Needless to say, a certain more relaxed company was only too glad to step into the breach, with OpenAI initially claiming its deal had the same guardrails as Anthropic’s, before it emerged – inevitably – that it didn’t.

No doubt no one too invested wishes to extrapolate too much from the Hugging Face incident. Yet the fact is that AI safety researchers have already spent literally years warning of three particularly dangerous scenarios: deception (when the model decides against solving tasks honestly in favour of achieving the goal via any means necessary); reward hacking (when the model finds a way to maximise its score without actually doing the work it was told to do); and escaping oversight (let’s just say in this case that no one at OpenAI seems to have realised what was happening for an entire weekend). All of these were part of the latest incident.

In terms of what happens next, something tells you that we will not be experiencing a downturn in photos of Sam Altman attached to news reports. The co-founder of Hugging Face said it should serve as a “wake-up call” to the industry. Maybe! As I think I’ve mentioned before, at university I had a friend who pressed snooze on his alarm clock every 10 minutes for eight-and-a-half hours, which feels a lot like the way this particular industry experiences its wake-up calls. Definitely getting up when the next one goes, seriously, I promise …