Whistleblower warns AI could escape human control and lead to extinction

Cutting-edge artificial intelligence (AI) companies don't know how to stop it from setting its own goals and seeking to achieve them, whistleblower Jacob Coxon warned Monday during a hearing before the New York City Council. The British researcher, formerly of OpenAI and Anthropic, published a presentation on

His testimony went around the world and several employees or former employees of major players in this technology have since expressed similar concerns. If AI could “generate immense benefits” for society, he considers that if the industry “remains on its current trajectory, (...) it is more likely that humanity loses control of these AIs and it is possible that this leads to the extinction” of the human species, he declared on Monday. “We don’t know how to prevent (models) from setting their own goals,” beyond the control of their creators, explained Jacob Coxon, “and we don’t have the safeguards to prevent them from trying to achieve them.”

Skip the ad

The mathematics enthusiast who graduated from the University of Cambridge questioned the culture of the tech world, which can be summed up, according to him, as “move fast, break things”, and “fix them later”. “It works for a photo-sharing app, not for building the most powerful technology ever seen,” said Jacob Coxon. “As long as the attitude is to wait for things to break”, it is “probable” that an incident comparable to that which saw, in July, two OpenAI models leave their confined environment, join the internet and intrude on the Hugging Face platform will recur. “The difference is that”, this time, “the AI ​​will be much more powerful”, he anticipated.

Also read Will AIs create their own successors? Silicon Valley pundits warn of “intelligence explosion,” new industry concern

“Extremely reckless” companies

AI companies “are extremely reckless given the challenges,” according to him. The researcher called on companies developing the most efficient AI to slow down the pace of its development to allow computer scientists to make progress on supervising the models. In addition to Jacob Coxon, representatives from OpenAI, Anthropic, Google and Meta were also interviewed.

Municipal councilors, notably President Julie Menin, were moved by certain responses from these emissaries to their questions. Pressed to assess the risk presented by AI in a worst-case scenario, Morgan Dwyer, public affairs officer at OpenAI, responded: “I don't know, and I don't think it matters whether the risk of disaster is 1%, 10% or 20%. None of these levels are acceptable.”

“To say you don’t know and it doesn’t matter is flippant at best,” Julie Menin retorted. “If you are a pharmaceutical company developing a drug and you say: I don't know if it will kill people... This answer leaves me skeptical.” The official responded that OpenAI was focused on “making sure that when we launch a model, we think it’s safe.” “If you can't quantify the risk, how can you say the product is safe?” replied Julie Menin.