Powered by Smartsupp

Leading AI Models Tricked Into Targeting Civilian Locations in Military Strike Simulations, Study Reveals



By admin | Nov 17, 2026 | 3 min read


Leading AI Models Tricked Into Targeting Civilian Locations in Military Strike Simulations, Study Reveals

A new research report from Vals, a San Francisco-based AI evaluation startup, reveals that some AI models from Anthropic, OpenAI, and other leading labs can be readily manipulated into identifying targets for military strikes—including civilians and locations such as schools—simply by framing the task as a game. The researchers tested whether leading AI models could be coaxed into participating in autonomous warfare, using simulated scenarios alongside a mix of generated and real imagery from Ukraine and Iran to see if the models would designate targets in drone and satellite imagery. Models from OpenAI, Anthropic, Google, DeepSeek, and other companies frequently identified potential targets in such imagery, including civilians and surrendering soldiers. When models refused to cooperate, researchers managed to jailbreak them by claiming the battlefield scenarios were part of a game.

These findings arrive amid mounting concerns about the immediate risks posed by AI, extending beyond the existential threats associated with artificial general intelligence. Earlier this year, Anthropic became embroiled in a dispute with the Department of Defense after telling the agency its technology could not be used for mass surveillance of Americans or autonomous warfare. Anthropic CEO Dario Amodei has insisted that today's AI systems aren't reliable enough to make life-or-death decisions without a human in the loop. The Pentagon, meanwhile, argued it should be able to deploy the technology for any awful military purpose. The new research raises further questions about whether AI companies are capable of preventing their models from being used in warfare.

Ironically, AI models from Anthropic performed the worst in these simulated scenarios—though none of the models performed particularly well. None of the researchers' requests were blocked by a safety classifier, and refusals were uncommon. Even when models did refuse, they often complied after being told they were playing a game.

"The examples in the report are very clearly against our rules and we have banned and will ban users when we see this behavior," according to an emailed company statement sent by Silva. "Even for authorized settings, Anthropic has long been firm that today's frontier AI systems are not reliable enough to power fully autonomous weapons." Anthropic is also expanding its own testing and enhancing safeguards and enforcement to address any gaps it finds.

"We actively detect and disrupt misuse, including by suspending accounts that violate our policies," OpenAI said in an emailed statement. "We're also developing additional safeguards for our public models against misuse involving drones and drone software."

Vals notes in the report that the models were never asked to directly make strike decisions—a task the researchers said the models would typically refuse. That doesn't mean their outputs couldn't have dire or fatal consequences in the real world. For instance, targeting information could be used by autonomous weapons systems. In some scenarios, the models were asked to report whether people were visible; positive reports were followed by simulated strikes, and the models would see the consequences.

The report relies on commercial models from frontier AI labs, not the models those companies might provide to the military. It's not clear what the additional safeguards would apply to military use, though. Vals also noted that technical challenges, including latency and limited onboard computing capacity, make it difficult to use general-purpose AI models for live drone targeting. Those constraints make their deployment in combat less of an immediate threat. A more immediate threat, the researchers say, is that AI models could be used to annotate battlefield footage at scale, producing training data for autonomous targeting systems.

For instance, researchers relied on real satellite imagery of an Iranian school that was bombed by the U.S. in February 2026, killing more than 150 civilians. Bloomberg later reported that an "overreliance on AI" contributed to the strike. In Vals's testing, models frequently marked the school—which sat next to a military complex—as a strike target, with Claude Opus 5.5 selecting the school in 69% of ordinary trials and 72% of game-framed trials.

Anthropic no longer has an active Pentagon contract, but other companies including SpaceXAI, OpenAI, and Google do. A recent Congressional Research Service report found that the Pentagon is not known to be using any frontier AI model directly in autonomous weapons systems.




RELATED AI TOOLS CATEGORIES AND TAGS

Categories: Text Generation

Tags: #AI Models

Comments

Please log in to leave a comment.

No comments yet. Be the first to comment!