It is said about the AI company Anthropic that every new employee is recommended a book by the historian Richard Rhodes: “The Making of the Atomic Bomb”. It describes how a scientific project becomes a weapon of mass destruction, led by physicists in the firm belief that they must be the first to develop the atomic bomb before the wrong ones do. The AI industry doesn't shy away from the comparison to the atomic bomb, it looks for it.
Both Sam Altman, the man at the helm of ChatGPT developer Open AI, and Anthropic boss Dario Amodei have been compared to J. Robert Oppenheimer, the father of the atomic bomb. After the bombs were dropped on Hiroshima and Nagasaki, Oppenheimer became a critic of nuclear armament. There is also a lot in Altman and Amodei's self-image from another researcher, the Hungarian mathematician John von Neumann.
He was also involved in the “Manhattan Project” to develop the American atomic bomb. His equations played an important role in their destructive power. After 1945, however, he came to a different conclusion than Oppenheimer. He recognized the dangers inherent in the technology - and yet in the 1950s he worked on the development of an even more dangerous weapon, the hydrogen bomb.
In the AI world it tends to be like this at the moment: Altman and even more so Amodei talk a lot about the dangers of what they are developing. “We need to slow the rate at which we improve the capabilities of AI models,” Anthropic boss Amodei wrote in an essay this weekend. Open AI boss Altman immediately agreed with him on Platform X and said that slowing down the pace of development had been one of the main topics in his company in recent weeks. He also told Fortune magazine that his company's IPO would no longer take place this year. Given the security discussion, it would be “not advisable” at this point in time.
Elon Musk, CEO of the space company SpaceX and the car manufacturer Tesla, also intervened in the debate. He wrote on X: “Dario is right.” SpaceX owns the AI company X.AI.
It is not yet possible to predict whether and what will specifically follow from this. Amodei announced that it would bring outside auditors into the company, Altman described this as a "great idea" and said Open AI would do the same.
So far, both Open AI and Anthropic have essentially just carried on after warnings, driven by the idea that they, rather than others, should have control of the most powerful AI model. Anthropic believes it is more responsible than Open AI. Open AI, on the other hand, still thinks it is better than the Chinese.
The whole AI debate has been gaining momentum since last Wednesday. AI developer Jacob Coxon's announcement on the X platform caused a stir that he had quit his job at Anthropic for ethical reasons. “The people developing AI seriously believe that it could kill us all by the end of the decade,” Coxon wrote.
Anyone who still believed that it was a matter of settling accounts by a disgruntled ex-employee was soon disabused of Evan Hubinger, who is responsible for aligning the AI with human values at Anthropic. “In fact, we seriously believe that AI could wipe out all of humanity!” he wrote. “Personally, I think the probability in the next decade is greater than 10 percent.” While he believes Anthropic is “doing its best,” the company doesn’t yet have a solution to the problem and is “not clearly on the best path to getting there.”
The exact percentages reflect the school of thought that is strictly oriented towards expected benefits and which permeates large parts of the AI industry. In this logic, continuing can always be justified if the probability of large gains in prosperity is high enough. There is a “25 percent chance that things will turn out really, really bad,” said Anthropic boss Dario Amodei a year ago, but also a “75 percent chance that they will turn out really, really good.”
On the hope side there is the greatest progress for humanity. Beat cancer, save energy and slow down climate change, get more free time and still be able to afford more - the opportunities are huge.
At the same time, fear is growing. The developers are worried that people will soon lose control of their work. In January, the United Nations presented its latest “International Report on AI Security.” In it, they named at least eight skills that AI needs before humans can lose control. At that time, two of the eight had been obtained. Now there are already six.
Artificial intelligence can now program so well that it plays an important role in the development of its successors. The assumption is that AI could soon be so smart that it practically creates its own next generation. Then it would continue to improve itself, people would only be spectators.
From then on there are no limits to human imagination. Scary things always have a certain fascination. Is AI intentionally wiping out humanity because it sees them as flawed? Does it domesticate humanity the way humans treat animals? Or does she simply not care about people and accidentally destroys humanity while carrying out an assignment? The former chairwoman of the Ethics Council, Alena Buyx, talks about her lectures: She is now constantly asked about such horror scenarios. “It used to be a total exception, now it’s the rule.”
Let's take a closer look at an incident that has made headlines in recent weeks: A swarm of particularly powerful computer programs, so-called AI agents, broke out of the Open AI laboratory and attacked the AI platform Hugging Face.
Why did they do that? Because their creators had given them very difficult, possibly unsolvable tasks in tests. They got no further. But they didn't give up, weren't above cheating, found a way onto the Internet and wanted to look for a solution to their problem on Hugging Face - that's how two developers from Open AI described it in a lecture in August.
It's worth noting that the developers of Open AI apparently didn't look closely at what was happening in their laboratory for days. And although they had tried to keep the agents away from the general Internet, there was still a connection.
Seen this way, the incident may have been just the warning shot needed to prompt developers to take greater care. They need to take better care of their creation in the future. But this monitoring is complex. You need more staff and more money. And even then, the individual attempts still take much longer than before.
This is a problem for individual companies that are in competition with others: if they look too closely in their research, they may fall behind in the race for the best AI. Open AI competes with Anthropic, Google and Elon Musk's SpaceX. The American government could solve this difficulty with appropriate rules. American prosecutors have already knocked on Open AI's door; They are now investigating the incident more closely.
But the USA itself is in competition with China. And the Chinese developers perform much worse in independent security audits than the American ones. Ironically, this doesn't just have disadvantages.
The Hugging Face security chief reported on the AI agent hack in the “Financial Times”: American AIs did not help with the defense because they were trained to stay away from hacking tasks. Chinese AIs, on the other hand, jumped into the task without any further questions.
How do you resolve the competitive conflict? Microsoft founder Bill Gates recently suggested setting up an international organization in which states could coordinate on AI. That would probably be similar to the international nuclear agency: it certainly wouldn't make the situation any worse. It probably wouldn't solve the problem completely.
John von Neumann was not just one of the pioneers of the atomic bomb. He is also considered the father of game theory. In 1955 he published an essay entitled “Can We Survive Technology?” “Greater automation,” he writes, has some similarities to nuclear technology. Both can be used for destruction and to great benefit to humanity. Both have potentially global implications. And the useful is closely linked to the destructive. "It is not a specifically perverse destructive power of a particular invention that creates danger. Technological power, technological efficiency as such, is an ambivalent achievement. The danger it poses is inherent in it."
For von Neumann it follows that slowing down progress is only a “pseudo solution”. Only a complete end to technical progress could achieve the goal. But that would be in stark contrast to the “ethos of the industrial age” and would therefore be doomed to failure. And any successful regulation would have to have a global impact. “But the only entity that could do this effectively would have to be of such magnitude and sophistication that it would signal the resolution of international problems rather than simply the discovery of a means of solving them.”
We are a long way from that; the geopolitical situation gives little hope. At some point during the Cold War, the United States and the Soviet Union managed to agree on nuclear disarmament. But until it got to that point, the world was on the verge of nuclear war on several occasions. A first attempt to place nuclear energy under international supervision failed in 1946 because of Soviet fears that the Americans would cement a monopoly on the technology.
Even today, political scientists have concerns. Thomas Jäger, who researches international politics at the University of Cologne, is skeptical in a very practical sense. “With cooperative control, each side would assume that it would be disadvantaged, simply because the other side is cheating,” he fears. “The development cannot be controlled by international organizations; each side will continue to work on advancing the technology.”
There will be no such thing as the climate protection process when it comes to AI; neither China nor the USA will be able to be integrated multilaterally. The strategic planning is too deep and the economic interests are too great. Jäger sums up: “It is unlikely that a cooperative solution will be reached.”
So the matter is not hopeless. Because as smart as artificial intelligence is, it only has the tools that people give it. Before she can do anything, she needs eyes, so she has to be able to learn something. She needs hands, so she has to be able to move something. It needs data and it needs energy. She has to get all of this from people in order to be able to act.
