In September, CNN reported that the United States came close to military confrontation with China. Why? According to the outlet, a military analyst using an AI chatbot to prepare an intelligence assessment of a Chinese vessel incorrectly concluded that the ship was transporting nuclear materials. American aircraft were reportedly in the air to intercept the ship by the time the mistake was caught.
An extensive discussion about how an AI model could make such a dangerous error followed. But as that deliberation dies down, it remains concerning how quickly an AI-assisted assessment may have moved through the chain of command and acquired the authority of vetted intelligence.
While we debate the impact of a future superintelligence on humanity, it is increasingly clear that AI will very soon enter the Situation Room, if it hasn’t already. As anyone who has been involved in a national-security crisis can tell you, there is often too much information, some of it contradictory, and not a lot of time or clear methods to sort through it all. AI can process volumes of intelligence inputs more than any human team, detect patterns and anomalies buried in the data, and make sense of complex information in remarkably little time. But what will it do to the president’s judgment?
Based on my experience studying crisis decision-making and assessing intelligence, I believe there is a real risk that AI will amplify leaders’ own biases, handing them back their own assumptions with the authority of objective analysis and the ease of a clear-cut outcome.
We tend to talk about AI as if it were wholly independent, a more rational intelligence system that can compensate for our weaknesses. But in reality, humans have built these AI models and trained them on data produced by humans. We then ask these AI models questions shaped by our own beliefs. The outcome is far from objective.
The flaws in human decision-making are well-known. Presidents and their advisers have always had to guard against confirmation bias, our tendency to interpret evidence in light of our pre-existing beliefs and to under-weigh or dismiss facts that point to the contrary. We know humans have selective attention, an instinct to hone in on facts that are more vivid and to overlook others. Presidents are not immune from these tendencies.
In my view, there is an even more insidious danger lurking within our AI use. Presidential decisions represent countless underlying judgments. Reasonable people can differ on each of the estimates, and those differences are part of what presidential judgment must resolve. Flattening them up into a probability assessment obscures more than it reveals because it does not show which indicators were seen as credible and which were ignored.
Consider the night of May 1, 2011. President Barack Obama had gathered his advisors to assess intelligence that indicated Osama Bin Laden was hiding in a compound in Abbottabad, Pakistan. If the intelligence was correct, he would send in American troops. But his trusted, experienced advisors, all looking at substantially the same data, offered individual estimates ranging from roughly 30% to 95%.
Now imagine a powerful AI system in that room and synthesized those assessments to give a 63.7% probability that bin Laden was there. We would certainly gain precision. But what would be lost? The number would not tell the president why there was such a marked difference in the assessments. It would not inform him of the competing assumptions on which those assessments were built, or how the implications of action versus inaction were weighed. But worst of all, it would not alert the president to the fact that there was disagreement at all. Such a “clean” number can make a question look resolved.
Now factor in that the CIA, DIA, State Department, military services, and other agencies will likely rely on the same foundation models built from overlapping data and sources. You could theoretically end up with 10 assessments pointing in the same direction but all derived from the same fundamental materials. The AI version of that risk is synthetic consensus: institutional agreement that looks independent but is partly generated by common analytical machinery and reproduced at scale.
But if the problems are well-known from human decision-making, so too are the strategies to counteract them. We can design our systems to act as benevolent antagonists and train our decision-makers to engage with them as such.
On the technology side, we need to build a system that talks back. AI used in the national security context should not be optimized just for speed of response. It should be built to help identify which evidence is credible; to generate the strongest competing explanations; to distinguish facts from inference and prediction; and to make uncertainty visible rather than manufacture precision. A system designed to be agreeable is dangerous. The points of friction and disagreement are often what’s most revealing and valuable for leaders making tough decisions.
On the human side, we need to ask better questions. That means noting priors before engaging AI, as well as taking a hard look at what one believes and at what confidence level. It means taking a second and third look at the framing and phrasing of the questions. Rather than seeking confirmation, we should explicitly ask for evidence that disproves our hypotheses and for ways of distinguishing between competing explanations. We should even be questioning whether we are focused on the right question.
