Barely a month after Jacob Coxon, a researcher specializing in training new AI models, left Anthropic for fear that the sector will not be able to control the technology it is creating, a new resignation has put the spotlight on technology companies and their focus on the growth of this technology. This time it is David Robinson, who led transparency work on the OpenAI security team.
Robinson has explained his reasons in an open letter published in The Atlantic and, while perhaps not as catastrophic as Coxon, he denounces the current path of the industry as "unacceptable."
In a reflection titled "I left OpenAI because its business culture is broken" (although the broken that Robinson uses can also be understood as "deeply deteriorated" or, simply, as not working), the manager ironizes the cliché that leaving an AI company begins to imply. "I join the parade of former colleagues, both from OpenAI and other leading companies in the sector, who have decided that the current path is unacceptable," he details.
Robinson explains that, like others before him, he believes that "the companies developing this technology are not being nearly careful enough." The nuance is that, in his case, he believes that the problem goes beyond regulations or legislation, but rather the business culture itself.
"The future depends on a wisdom that Silicon Valley lacks," the former manager goes so far as to say. It refers to the knowledge needed to understand “how to manage dangerous technologies,” but also, “more fundamentally,” wisdom “about what it means to care for people.” And here he charges against the sector, especially technology managers, as he believes that the current context "requires a degree of humility that is not natural for those who have succeeded thanks to extreme self-confidence."
In this sense, Robinson recalls that to achieve the technological achievements of recent years, it has been necessary to pay "an enormous cost", with unaffordable work rates. "The approach to safety that emerges from such a culture is based on unfettered optimism about the ability to solve problems as they arise," he says. OpenAI itself, it illustrates, has been solving problems as it detected them, but this approach "guarantees periodic failures" and, furthermore, "the magnitude of these failures increases as the systems acquire more capabilities." Here Robinson mentions the famous Hugging Face incident, when a series of agents autonomously hacked the platform. The company, he details, responded by improving security. "However, even after those changes, OpenAI reported that its security controls failed again when a model in training bypassed internet access restrictions: a monitoring system alerted human staff, but did not automatically shut down the model as it should have done," he explains.
Robinson doesn't just point to his former company. Anthropic had a similar case, for example, but the researcher believes the problem is that "these types of errors are common in the industry, given the speed and flexibility with which people work." "An environment where things like this can happen is not a suitable place to develop artificial minds that could become more intelligent than us and that may not do what we want them to do," he insists before warning that "people will not be safe if we depend on individual heroic acts to solve problems after they occur."
Still, Robinson acknowledges that "this technology can be useful and valuable" and that his former colleagues "are smart, work hard and try to make good decisions," a somewhat more optimistic view than Coxon's. In any case, he considers that two changes are urgently needed. First, “AI companies need to lean more on the security expertise that already exists in other sectors.” On the other hand, he believes that before creating "systems significantly more capable than today's ones," we need to ensure that the most advanced models and their successors "will make safe decisions when we are not watching them." The sector, he insists, should operate like nuclear power plants or busy airports do, "with layers of redundancy and careful, laborious planning, so that occasional and inevitable human errors do not open the door to disaster." And right now, "AI companies don't know how to do it, but other people do."
