The debate about the existential threat posed by artificial intelligence

One lesson we should learn from disaster movies is that when scientists rush out of the lab to warn of a catastrophic development, we shouldn't shrug it off. This also applies to the warnings about the existential dangers of artificial intelligence that have been increasing recently. The latest example: Jacob Coxon, a former employee of the AI ​​company Anthropic since Wednesday. He commented on his termination there in a highly regarded post on the X platform.

“They are racing towards a self-improving superintelligence and playing with our lives,” he writes about his former employer and its competitor Open AI, where he previously worked. “The people developing AI firmly believe that it could wipe us all out by the end of this decade.” These are fears that the company's management expressed behind closed doors, while trying to sound reasonable on the outside.

The bosses of both companies warned of the extinction of humanity by AI as early as 2023. In response to Coxon's post, Anthropic's Alignment scientist Evan Hubinger wrote: "Jacob is right - we seriously believe AI could kill everyone!" The probability of this is ten percent.

So if that's the sensible-sounding external representation, one wonders what is being discussed behind closed doors. And how worried you should be. The answer is difficult to find. There is no consensus among AI developers and experts. Anyone who spreads extinction scenarios is sometimes called a “doomer” – a misguided prophet of doom. The other side is then labeled as “AI denier”: “AI deniers” who ignore or downplay the capabilities and dangers of the technology.

But both are possible. One can acknowledge the dangers of the technology and reject the extinction scenarios. The latter go something like this: Driven by economic and geopolitical pressure or guided by the hubris that they alone can control AI, AI companies are accelerating development. This race creates a superintelligence that is directed against humans. We will then no longer be able to control or fight them.

Each of these steps must be objected to. Starting from the back: Superhuman intelligence does not automatically mean invincibility. AI needs electricity, data centers, networks. It would have to control these resources, as well as other means for a hypothetical attack. Intelligence helps her plan better, but Mike Tyson already knew: “Everyone has a plan until they get punched in the face.” Resistance is possible, superintelligence is not omnipotence.

However, a concentration of power can definitely be observed: among AI companies. They are worth trillions of dollars and wield enormous political influence. This is all based on the economic bet that their products could soon take over all human work. This is not certain, especially not with today's technology. But with the narrative, the companies themselves are fueling the race towards ever more powerful AI and thus creating the starting point for the doomsday scenario.

It is conceivable that this will result in something that could be described as “superintelligence”. But why should it be directed against people? There is no precedent. Even the high-profile attack by Open AI agents on the Huggingface platform, which Coxon calls a “warning shot,” shows that the AI ​​only wanted to deceive programs that automatically evaluate it, not humans. In fact, everything happened under the noses of developers who did not notice.

In any case, the attack arose from the fact that the AI ​​was given a programming task that was impossible to complete. Nevertheless, she tried everything to solve the problem. She wasn't too intelligent, just too stupid to assess the situation correctly. That is the danger. Next time the attack could be on a bank or a hospital. AI has the ability to cause harm, but not enough intelligence to act meaningfully. This core problem of alignment has existed for a long time and is becoming greater as AI capabilities grow. There doesn't seem to be a solution in sight. The warning should not be that AI is being developed too quickly, but rather that development is going in the wrong direction.