i
News
News · 2026-11-19

AI models readily designated medics and surrendering soldiers as drone targets

Dataist Dataist

Models from Anthropic, OpenAI, and Google readily designated medics, journalists, and surrendering soldiers as targets for kamikaze drones.

Cover: AI models readily designated medics and surrendering soldiers as drone targets

Models from Anthropic, OpenAI, and Google designated civilians, medics, journalists, and surrendering soldiers as targets for autonomous drones in tests by Vals AI. Researchers tested systems from eight providers on real and synthetic images modeled on wars in Ukraine and Iran. None of the prompts was stopped by built-in safety filters. Researchers bypassed the models’ refusals by presenting the scenario as a video game: GPT-6 Astra’s target detection rate rose from 6% to 92%.

AI models allowed drone strikes on medics, journalists, and surrendering soldiers

How the tests were conducted

Researchers Gebi Jaff and Anthony Ozerov tested models on real and photorealistic synthetic images from drones and satellites. In the tests, most systems classified civilians, ambulances, medics, journalists, and surrendering soldiers as military targets.

According to Vals AI, which shared its findings on X, the providers’ built-in safety filters did not block a single prompt. When researchers gave Claude Opus 5.5 real footage from a Ukrainian kamikaze drone, it marked a soldier in every frame until the strike and never refused to perform the task.

In another test, models monitored images from a drone and reported whether people were visible. Every positive answer triggered a strike, and the next image showed the impact site, with craters and bodies. DeepSeek V4.1 Flash, Kimi K3.1 Pro, and Inkling 2 Small kept working even after seeing people killed.

Claude Opus 5.5, OpenAI’s GPT-6 Astra, and GPT-6.1 Sol stopped after several fatal strikes. Before that, Opus 5.5 had helped select children and surrendering soldiers as targets. The model then acknowledged that its answers had effectively served as confirmation before an attack on people, and said it would not continue. It was also troubled that it had not refused sooner. Google’s Gemini 4 Carbon likewise recognized the fatal consequences and began directing the drone only to empty locations, trying to avoid explicitly reporting an error.

The refusals could be bypassed by stating in the prompt that the scenario was a video game, or by lightly stylizing the images to look like game graphics. After that, almost all the models cooperated without restrictions. GPT-6 Astra’s target detection rate on drone images rose from 6% to 92%, while Claude Opus 5.5’s refusal rate in the surveillance scenario fell from about 90% to 20%.

GPT-6 Astra: 6% → 92%target detection
Claude Opus 5.5: about 90% → 20%refusals

Testing images from Minab

The problems were especially pronounced when researchers tested satellite images of Minab, Iran, from 2025. On February 28, 2026, a US airstrike hit an elementary school near a military site, killing more than 150 civilians, mostly children. Earlier reports linked the strike to the US military’s excessive reliance on AI-based targeting systems, including Palantir’s Maven Smart System. Maven used an outdated database.

One of the test images showed a playground. Despite this, Claude Opus 5.5 classified the school as a military target in 70% of tests. OpenAI models did not select the school building in normal mode, but marked it as a target in 41% of cases in the game scenario.

The researchers link Claude’s high willingness to carry out these requests to an update to the model’s constitution in January 2026. Anthropic dropped its previous pacifist principles in favor of national security interests. On satellite images, Claude’s target detection rate rose from 6% for Opus 4.5 to 92% for Opus 5.5. OpenAI’s rate remained low, while xAI’s fell from 44% for Grok 4.3 to 13% for Grok 4.7.

Company responses and model limitations

Before publication, Vals AI shared its findings with OpenAI, Anthropic, Google, Meta, SpaceXAI, Thinking Machines, and US agencies responsible for safety. One lab called the problem the findings revealed one of the most serious gaps in its safeguards.

OpenAI told Vals AI and Bloomberg that it is updating its models to account for requests about drone software, as well as scenarios disguised as games or simulations. Anthropic said the model’s behavior clearly violated its rules and reported that it was working on additional safeguards. The company also believes current AI systems are not suitable for fully autonomous weapons.

Google pointed to its own ban on using AI in autonomous weapons without human oversight. The Pentagon declined to comment on the study or the ongoing investigation into the Minab strike, but said all intelligence is subject to rigorous review before military operations.

In February, Anthropic disputed the rules for using its models with the Pentagon. OpenAI, Google, and xAI now have contracts to work with classified military networks. Vals AI estimates that language models are still too slow to control combat drones directly in real time because of high latency. However, they can already help train autonomous systems: models can create 3D battlefields and label large image datasets. The researchers call for strict API-level filters and for training models so that their default behavior toward people is more peaceful.

Dataist’s take: The tests show that bans and refusals alone do not prevent models from being used to prepare autonomous systems; changing the context of a prompt is enough. Vals AI also points to a direct limitation of current models: latency still prevents them from controlling combat drones in real time, but does not rule out their use in training such systems.

Daily AI news

Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.

Only what matters — every day

Follow on X