This spring, U.S. officials made a disturbing discovery while military aircraft were already airborne: the intelligence behind an armed operation targeting a Chinese vessel had been fabricated by an AI chatbot. According to CNN, the operation was called off at the final moment, barely preventing what could have escalated into a confrontation with China.

The incident highlights a deepening worry among military leaders and independent analysts: as decision-makers increasingly rely on AI, the mistakes these systems generate can move up the chain of command before anyone catches them.

The intelligence report, which was circulating during the war with Iran, claimed the vessel was transporting components for a nuclear weapons program. The false information traced back to a Special Operations Command analyst who had asked an AI chatbot to combine open-source data with classified signals intelligence. The chatbot incorrectly identified the ship's cargo manifest. The analyst then ran the tool a second time to turn the flawed findings into a formal-looking summary, which was subsequently distributed through command channels.

This near-disaster unfolded as the U.S. military pushes hard to incorporate AI to speed up decision-making and preserve its advantage over China. The Pentagon has characterized AI as providing a major edge in accelerating its kill chain so commanders can act within the necessary timeframe. Yet the very speed that makes AI appealing may also permit hallucinations to slip through with inadequate human review.

"It's important for service members to understand the uncertainty inherent to LLMs," said Jake Steckler, a research scholar at GovAI and a U.S. veteran. "But it's especially critical for any decisions that could lead to use of force, like targeting, intelligence analysis, or operational planning. There are life and death consequences for those decisions."

Even so, Steckler argues the incident should be treated as a reason to build in more safeguards for AI, not to steer clear of it. "These tools can be useful in the right contexts and with the right safeguards in place," he said. "But prioritizing adoption speed over all else will likely lead to incidents that only make service members lose trust in these systems, which ultimately is only going to slow adoption."