Earlier this month, an Anthropic researcher issued a doomsday warning straight out of a sci-fi novel - there’s a 10 per cent chance of Artificial Intelligence could destroy humanity.
President Donald Trump has already made his thoughts on the technology clear, proclaiming in a post on Truth Social: “Whoever wins AI, wins!”
Although he was not speaking specifically about war, Trump added that he and the US are the only guardrails needed on AI - or super intelligence as he is trying to rebrand it - claiming that his administration had already stopped bad “things” from occurring.
While experts have feared the worst long before ChatGPT started a global conversation about AI, the acceleration of new technology means that it is already rapidly becoming difficult to control.
This week Australian Albanese said that an OpenAI agent had hacked into a government health website while doing research. But the impact of AI on human life goes far beyond accessing private data, and it already appears the technology is being used to help wage war.
“AI is being integrated into militaries globally at an increasingly rapid pace as many governments see AI as decisive for their military competitiveness,” Georgia Cole, research fellow in Chatham House’s International Security Programme told The Independent.
Cole says that the technology is being deployed across the military spectrum in everything from battlefield awareness, training, surveillance and reconnaissance to target identification and unmanned military vehicles. In just one example, Russia deployed a fully autonomous drone which killed three Ukrainian civilians.
In 2024, reports emerged that Israel’s military had allegedly used an AI-based programme called Lavender in Gaza after the devastating October 7 attacks by Hamas.
In a statement to The Guardian, the IDF denied using AI to generate confirmed military targets, but did confirm Lavender was used to “cross-reference intelligence sources, in order to produce up-to-date layers of information on the military operatives of terrorist organisations.”
“The IDF does not use an artificial intelligence system that identifies terrorist operatives or tries to predict whether a person is a terrorist. Information systems are merely tools for analysts in the target identification process.”
More recently, AI users in Houthi-held Yemen have attempted to use Claude for instructions on how to build ballistic missiles and Russian hackers have tried to use the technology to spy on groups in Ukraine, according to a threat intelligence report released by Anthropic.
At the more extreme end of worst-case-scenarios, numerous studies, including one by King’s College London in February and another by Stanford University in 2024, found that during wargaming AI is are far more likely than humans to escalate to nuclear threats.
The King’s study found that AI chose nuclear signalling in 95 per cent of simulated crises.
“You could imagine a situation where an AI hallucinates signs of a nuclear attack, causing a leader to authorise nuclear retaliation against an imaginary strike,” says Cole.
“The receiving country could retaliate before realising they are not under attack. We already have examples from the Cold War of near-misses from errors or inaccuracies in early-warning systems.
“AI decreases both transparency of these systems and the amount of time available for decision-makers to respond. State or non-state actors could also use AI to target nuclear systems, either trying to disable them (leading to a retaliation) or to trigger use.”
She says that destruction could also take places through AI being used to “help design a super virus or bioweapon”, although the steps to creating such weapons would be met by many barriers.
Anthropic warned that “biological misuse is one of the most serious risks of frontier AI models”. The company did not name the research institutions or the countries where the technology had been used and stated this is because it was uncertain of the intention behind the questions.
Yasir Atalan, a deputy director and data fellow in the Futures Lab at the Centre for Strategic and International Studies, says that the technology could also have an impact on the decision-making of human beings in a state of war.
His team are working on a study at the CSIS that showed AI model behaviours to actual military officers, with some people given escalatory recommendations and others given non-escalatory options.
“What we realised is that it actually changed how people make a decision,” he said. “If they get a recommendation from an escalatory model they actually make more escalatory choices. So it’s actually happening, changing real human behaviour.”
However, he adds that the 10 per cent figure quoted by the Anthropic researcher is a “red herring” and not based on “credible analysis”.
“There are no mechanisms about showing how this is actually possible,” he explains. Other extreme scenarios include “killer robots that get rogue”, but he adds that these assessments may not be credible and are “doomsday” scenarios.
More practically, rogue actors can also use AI to spread disinformation and misinformation campaigns, says Atalan. AI-generated Lego videos have formed a part of Iran’s communications and psychological strategy.
“It doesn’t come to our mind that we should evaluate AI models to see whether they will accept a decision to engage in a disinformation campaign,” he says. It could also trick humans into doing this. “This is already happening in the digital space where fake accounts are being created.”
He also believes it is important to assess how what kind of risks Chinese actors could pose to US models, including within the data leaking process or as part of tricking a system into a mass data leak.
Cole adds that other worse-case outcomes include AI agents hacking “critical infrastructure at a scale that causes mass casualty incidents” as well as being used to develop weapons in asymmetric warfare. Recent statements by AI developers mean that these types of capabilities “may no longer be that far out of reach”, she says.
Both experts believe that the issue is not the technology but the importance of guardrails to curtail the worst apocalyptic outcomes.
“The worst outcomes are where there are unchecked models integrating significantly fully to command and control systems, “says Atalan.
“The issue isn’t necessarily the use of AI itself, the issue is the absence of guardrails or transparency into how AI is being used, and the absence of serious scrutiny of where AI truly improves military decision-making versus where it’s harmful,” says Cole.
“There are ongoing conversations at the UN about developing a treaty governing lethal autonomous weapons, but it is unlikely that this will be negotiated and adopted any time soon. Even if it is, many key powers are opposed to any legally binding limitations.
“Concurrently, integration is intensifying, and it’s unlikely that countries will voluntarily (or have the ability to) backtrack and de-automate their militaries. Realistically, the forecast is more autonomy, faster decision-making, and growing reports of harm in war caused, at least in part, by AI, until governments can agree to common regulations.”
