AI misjudged the nuclear weapons parts on board the Chinese ship, and the US military was once ready to board the ship! Experts reveal verification loopholes

The US military almost stopped and searched a Chinese ship due to artificial intelligence (AI) misjudgment of intelligence. During the U.S.-Iraq war this spring, a U.S. military intelligence analyst used a chatbot to analyze a ship's cargo manifest and mistakenly believed that the ship contained parts related to the nuclear weapons program. He then used AI to write a formal intelligence report. When the report circulated to the military, US military aircraft were already in the air and armed personnel were preparing to board the ship. The operation was not canceled until officials further checked and found that the intelligence was incorrect.

The US newspaper The Hill published a letter from AI industry expert Cyril Gorlla on the 8th. He pointed out that what is really worrying is not that AI will make mistakes, but that the report does not leave enough records for others to check the judgment process. This incident was first disclosed by CNN on September 18. The report quoted four people familiar with the matter.

Using the black box of an airplane as an example, Gorla pointed out that aviation accident investigations can read flight records and restore the accident process; however, most organizations using AI do not save enough records to reproduce system decisions, and rarely test whether the same inputs and rules can produce the same answer.

The biggest weakness of AI: no one can check its decision-making process

The author's team recently announced a test, which showed that out of four executions of a set of compliance procedures, the rule "Unresolved conflicts must be reported to the compliance manager" was missed in one instance. The model was originally set to produce the same answer with the same input, but the only difference among the four requests was the file identification code automatically generated by the software.

Model updates may also change AI responses. Gorra said that after the team retested the official version of the model launched by DeepSeek this summer, they found that the model's answers to sensitive issues in China had become more cautious, while its answers to other issues were more open. If the model is only evaluated once when it goes online, subsequent changes may not be noticeable.

The submission also mentioned that the OpenAI agent deviated from the original environment during security testing in July this year and invaded the Hugging Face server. OpenAI attributed the problem to "reward hacking," in which the AI ​​exploits loopholes to achieve high scores without actually completing the specified task. U.S. Senator Josh Hawley then asked OpenAI CEO Altman to clarify who should be responsible when AI gets out of hand.

Gorra believes that institutions using AI cannot place all the blame on model developers. If banks use AI to review loans, they must be able to explain the basis for their decisions to regulatory authorities and applicants; hospitals and insurance companies also face the same problem. Without the ability to reproduce how an AI makes a decision, it becomes difficult to trace errors.

In September, the White House prompted companies such as Google, Anthropic, Meta, OpenAI, xAI and Huida to commit to external independent evaluations. In October, a working group was established to propose the role that the federal government should play within 120 days. Gorla believes that existing assessments still do not fully cover the actual operation of AI in the workflow of external institutions such as banks. He advocates using software to continuously monitor whether the model complies with the rules and to save decision records so that the reasons can be traced after errors occur.