This spring, during the war with Iran, armed members of the US military were preparing to board a Chinese ship in the Middle East. Military planes were in the air. The intelligence report that put them there said the vessel was carrying components of a nuclear weapons program, and any boarding of a Chinese ship risked pulling the two countries into armed conflict. Just before the operation, officials looked harder at the report and found it had been generated with the help of a chatbot that had misidentified the ship's cargo. CNN could not establish what the cargo actually was. One source called the report "entirely false" and said it "almost started a war."
The mechanics are worth following, because they are ordinary. A special operations command analyst queried a chatbot about intelligence reporting on the ship's manifest, material that originated with US Special Operations Command Pacific in Hawaii. The bot fused open-source intelligence with secret signals intelligence held in government systems and reached its conclusion about what the ship carried. The analyst then used AI again to package that conclusion into a standard intelligence report, the kind military officials trust, and disseminated it.
That second step is where the damage was done. The hallucination was the model doing what these models do: producing a confident, fluent answer whether or not the underlying data supports it. The failure mode is known, it is not rare, and sources say this was not an isolated case across the intelligence community. A system that uses a language model to fuse sources should treat a wrong answer as an expected output. Instead the report went through a packaging step that turned it into a standard intelligence report, the kind military officials trust. Downstream, there was no agreed standard for verifying what these tools produce.
The surrounding structure was not set up to catch an error like this. US officials describe the military's AI adoption as decentralized: different parts of the government run different tools under different orders and different safety standards, with no single standard for verifying what the tools produce. Reliability varies widely from system to system. A former senior official familiar with the systems called the internal tools "mostly just copies of the commercial stuff wearing lipstick." If that holds, the government is inheriting the behavior of commercial chatbots and stacking classified data, targeting decisions, and a trusted report format on top.
The direction of travel widens the gap. In January, Defense Secretary Pete Hegseth released an "Artificial Intelligence Acceleration Strategy" to speed adoption; the announcing memo described putting AI models "directly in the hands of our three million civilian and military personnel, at all classification levels." Hegseth said the department would "unleash experimentation, eliminate bureaucratic barriers." The stated reason is speed. AI is meant to help decide which targets to strike and which assets to move faster, on the argument that the US cannot fall behind China. Sources said targeting is one of the areas ramping up, and that there is no real guidance for how a human in the loop is supposed to prevent civilian casualties or fratricide.
Speed is where the risk compounds. Older intelligence officials, including ones who support military AI, told CNN the tools pressure analysts to produce and disseminate faster, which opens the door to mistakes. Younger analysts, several sources said, grew up on these tools and are more likely to trust them without checking. One source summed it up: "AI allows you to get to a bad idea faster."
If you run a system like this, the thing to fix is provenance. The boarding party nearly moved because the hallucination reached them in a format that told them to trust it. Provenance has to survive the packaging step. A report that came from an AI fusion of sources should carry that fact, and the specific sources it leaned on, all the way to whoever acts on it, so the reader applies the scrutiny the format would otherwise suppress. A human in the loop only helps if the loop hands them something to check. Here the deeper look happened late, just before the operation.
Top comments (0)