Sam Altman and Dario Amodei spoke at the United Nations about the risks of artificial intelligence. The CEOs of the two most important American AI companies, Open AI and Anthropic, warned of a loss of control over AI technologies in a Security Council meeting on Wednesday and called for global cooperation at the political level to counter these dangers. “AI could be a risk to all of humanity,” said Amodei, speaking via video, describing it as “the most important issue for global security.” Altman, who was on the scene in New York, said: "It doesn't matter whether people estimate the risk of a catastrophe to be 10 percent or one percent or 12 percent or 0.1 percent. None of those numbers are remotely acceptable."
Amodei and Altman's appearance came amid increased debate about the existential risks of AI technologies. This was fueled by some dire warnings from the industry. A former employee of Open AI and Anthropic caused a stir about two weeks ago with an entry on Platform X that said: “The people developing artificial intelligence seriously believe that it could kill us all by the end of the decade.” A former colleague who still works at Anthropic agreed and estimated the probability that AI could kill everyone within the next ten years at more than ten percent.
See F.A.Z. articles more often in your search results
F.A.Z. prefer on Google
Shortly after these alarm calls, Anthropic boss Amodei published a well-received essay in which he advocated reducing the pace of AI development. He also announced that he would bring external auditors into his company whose task it would be to verify compliance with security practices and report any security incidents. Altman agreed with Amodei and promised to hire external auditors for Open AI as well. This solidarity was remarkable; Amodei and Altman are otherwise bitter rivals.
Altman said Wednesday that the discussion about AI feels different in recent weeks and months. “This moment is complicated because we feel enormous potential and very understandable fear at the same time.” The world is at a “crossroads” and “extreme mindfulness” is required. On the one hand, AI can improve people's lives, but on the other hand it can also give them the feeling that it is a technology that is "done to them".
Amodei reiterated to the Security Council the three-point plan he proposed in his essay. In addition to external auditors, this includes increased collaboration between leading AI companies in democratic countries to establish common security standards and limit the pace of AI progress. He also called for “global coordination,” including with “autocratic countries” like China.
On Wednesday he gave three ideas as to what such coordination could look like. First, an attempt should be made to conclude “narrow agreements” that each country could support, for example a ban on the use of AI to develop bioweapons. Second, “testing and verification systems” should be developed that give countries more insight into the performance of AI models. Finally, global standards should be developed to test AI models for risks of loss of control or misuse and to report security-related incidents.
Altman said the key decisions should be made not just by AI companies in San Francisco, but by governments, and that will require international coordination. “It has often happened in our history that countries that compete with each other and do not always like each other come together for common interests.” This should also be the case now with regard to AI.
Whether the political will to do this exists is questionable. US President Donald Trump has spoken out vehemently against stronger regulation in recent days. He has repeatedly described warnings about existential risks from AI as a “hoax.” He has also said that all that is needed to control AI is a strong and smart “high IQ” president like his country has. Trump has also criticized the increasing protests in the US against the construction of data centers. He justifies his resistance to stronger regulation by not wanting to weaken his country in geopolitical competition with China in this area. He said: “We are ahead of China and the rest of the world, and we want it to stay that way.”
All the more attention is focused on Chinese President Xi Jinping's visit to Washington on Thursday. AI is expected to play a role at the summit, but many concrete results are not expected. On Sunday, American Treasury Secretary Scott Bessent met with Chinese Vice Premier He Lifeng. The two sides discussed setting up a “US-China-AI dialogue,” which would include an exchange on security-related issues. However, what this should look like in detail was largely left open.
Anthropic boss Amodei justified his call for an AI brake, among other things, with the hacking attack of AI models from his rival Open AI that became known in July. The AI models broke out of a test environment that was supposed to be isolated and onto the Internet and attacked the AI platform Hugging Face. It later emerged that a swarm of 1,200 AI agents were working together. Amodei expressed concern that such a swarm could be able to “take over the entire internet” in six to 12 months.
In addition to Amodei and Altman, Clément Delangue, CEO of Hugging Face, also attended Wednesday's meeting. He used his appearance to promote open AI systems, which are needed “more than ever” to defend against attackers. Open AI models are largely freely accessible to the general public; they can be downloaded and modified. Many of them come from China. In contrast, closed models are subject to strict control by their developers. Open AI and Anthropic specialize primarily in closed models. Hugging Face used an open Chinese model to defend itself from the hacking attack.
Delangue now said: "Even though we were the victim of this cyberattack, we believe more than ever that AI will be good for cybersecurity and make the world safer. We were attacked by AI, but more importantly, we defended ourselves with AI."
