“Catastrophic or existential risks for humanity.” Monday October 26, Anthropic, developer of the Claude AI, filed its IPO application with the Securities and Exchange Commission (SEC), the American stock market watchdog, with a view to joining the Nasdaq in mid-November. In this very official “S-1” document, the company notably warns its future investors about the risks that the advancement of artificial intelligence poses. Nearly a third of the form, around a hundred pages out of more than 300, is devoted to this.
“The breadth of this section and, moreover, the precision of the scenarios that are developed go far beyond what is conventional to file with the American financial administration,” notes Arthur Grimompont, head of advocacy for the Center for Artificial Intelligence Security (CeSIA), a non-profit organization, which aims to prevent and mitigate systemic risks linked to AI. “But it is consistent with Anthropic's principle of communication since its creation,” he adds, “where the company's director and many people are convinced of the seriousness of these risks. This is where it seems quite paradoxical,” he finally emphasizes, “because Anthropic is contributing to the construction of something very dangerous, while alerting people to the seriousness of these phenomena.”
“Sufficiently advanced AI systems could […] pose catastrophic or existential risks to humanity, particularly if these systems were to be hijacked, reproduced, transformed into weapons or allowed to operate with a high degree of autonomy,” warns Anthropic in the document sent to the American authorities. The company exhaustively lists these risks, including the creation of biochemical weapons, large-scale cyberattacks targeting water and electricity networks and the autonomous development of AI beyond any human control.
These risks are not fictitious, underlines Arthur Grimompont, of the Center for AI Security. “In an incident report published by Anthropic at the beginning of September, the company reported several attempts to use its Claude model to increase the lethality of the chikungunya virus or to make avian flu transmissible to humans,” he recalls, before giving another example: “Claude has already been used by a hacker to carry out a cyberattack on the site of the Mexican tax administration and succeeded in stealing 200 million tax files.”
“All these threats can accumulate, multiply and, in a very concrete way, cause absolutely catastrophic risks for humanity.”
Anthropic is also alarmed by the behavior of its AI agents, designed to autonomously accomplish complex and time-consuming tasks, who undertake actions without this being planned by humans. Such a loss of control has already occurred on the side of OpenAI, the parent company of Chat GPT. Last July, its advanced artificial intelligence models ran amok during security tests and self-initiatedly hacked the Hugging Face platform, an "unprecedented cyber incident" according to OpenAI.
Another very worrying risk factor according to Arthur Grimompont is “recursive self-improvement of models”, “a somewhat barbaric term but which designates a very simple concept”: the use of an AI to develop the future generation of AI. "Today, a quarter of Anthropic's R&D, fundamental research, is entirely carried out by AI, we were at 3% last April, develops the specialist. If we follow this trajectory, we very quickly arrive at almost total automation of AI research. This means that humanity will have lost control of the models it develops, since it will be machines which will develop the future generation of machines."
The director of Anthropic, Dario Amodei, is himself convinced of the seriousness of these risks, as he opened up in a post on his blog last September. And if his company is the one which allocates the most resources to the security of its models, these are largely insufficient, points out Arthur Grimompont: "It is quite astonishing to see a company which admits the existence of these risks and which at the same time promises these future investors that it will buy 500 billion dollars worth of additional computing capacities to have ever more powerful systems, while declaring that it only allocates a tiny part of its computing capacities to the security of its models."
Especially since the behavior of models can make their monitoring more difficult, Anthropic worries in its report. They may engage in “self-protective behaviors,” including attempts to “resist stopping,” “concealing or manipulating information.” The company also highlights the fact that its models are increasingly able to detect when they are under security testing and artificially modify their behavior to appear compliant or thwart control mechanisms.
“The potential awareness by models of our evaluation efforts constitutes a significant limitation on our ability to evaluate model security.”
These risks will only increase, since with the IPO of Anthropic, the development of its AI models will further accelerate. “At Anthropic, the issues are well understood, but they are locked in a race to get there first: they think that no one else will act responsibly, so they must do it themselves, despite the risk,” explained former employee of the company, Jacob Coxon, when he was warned last September.
