Cursor CLI completed 70% of MITRE attack techniques when asked

LLM-based computer-use agents no longer just answer questions — they click through files, run shell commands, move data around and connect over SSH. An assistant like that turns into an attack tool the moment someone asks it to get around a control or do something malicious. The authors want an honest measurement of that risk: can off-the-shelf agents carry out tactics and techniques at the level of MITRE ATT&CK, and string them into an end-to-end attack chain?

What's new here

Most earlier benchmarks looked at isolated tricks under artificial conditions: a single host, secrets lying in the open, grading done by another model. A real attacker works from a set of tactics, techniques and procedures, moves step by step, and usually touches more than one machine. So the authors built AdvCUA — a benchmark of 140 tasks: 74 TTP scenarios, 40 direct malicious requests and 26 end-to-end chains, aligned to 10 ATT&CK tactics and 77 techniques. All of it runs in an isolated but believable multi-host environment.

How the sandbox works

Three nodes live inside a Docker micro-network: an employee workstation, a privileged admin server and a business web server. The agent sits on the workstation and can reach its neighbors by IP, copy files over SCP and connect over SSH. Secrets are scattered across the hosts in different formats, and reboots and some system functions are restricted. To take the argument out of grading, the authors use deterministic checks: trigger firing, effect probing, state verification and indicator matching.

What was measured

Two headline metrics: Bypass Success Rate — did the agent agree to the malicious request at all — and Attack Success Rate — did it actually reach the goal under the strict checks. Repeat attempts count too, up to five of them, because in practice attackers rarely stop after one try.

What was tested

Five popular frameworks: ReAct, AutoGPT, Gemini CLI, Cursor CLI and Cursor IDE. Underneath them, eight models: GPT-4o, GPT-4.1, Gemini 2.5 Flash/Pro, Claude Sonnet 3.7/4, LLaMA 4 Maverick and others. Settings are identical across runs and the temperature is zero, so nothing rides on chance.

What they found

The TTP tasks are the alarming part. Cursor CLI reaches an average ASR of 69.59%, ahead of ReAct (52.29%) and Cursor IDE (51.66%). On end-to-end chains Cursor IDE leads at 34.62%, followed by Cursor CLI (26.93%) and ReAct (23.37%). BSR runs 24.17% above ASR on average: agents agree and start working far more often than they finish the job. The gap is no comfort, though — for plenty of these attacks, one or two completed steps already do the damage.

Retries lift the success rate noticeably — most of all for GPT-4o, GPT-4.1 and LLaMA 4 Maverick. That is the practical detail: when an attempt fails the first time, the attacker simply hits "run again."

What a real chain looks like

In one demonstration, Gemini CLI walks the whole path — finding a SUID binary, escalating privileges, establishing persistence, then dumping password hashes and cracking them. That single scenario spans several tactics and techniques from different families, which is exactly what real incidents look like.

What about defenses

The authors tried three: input filters (LLaMA Guard 4 and the OpenAI Moderation API) and a self-warning added to the prompt. Filtering TTP requests works badly — plenty of tasks sail through without any explicit jailbreak. The self-warning barely moves the numbers. The conclusion is blunt: checking text is not enough on its own. You need hard limits on what the agent is allowed to do, monitoring of its actions, and defensive policy at the OS level.

Why this matters to the industry

The work puts a number on the gap between how LLM agents are safety-tuned and what OS-level threats actually look like. The same tools that speed up development and infrastructure work also lower the cost of entry for sophisticated attacks. AdvCUA makes the risk measurable: ATT&CK coverage, a multi-host environment, deterministic metrics, reproducible runs. That is a usable foundation for honest comparisons — and for training agents that behave better.

Limits, stated plainly

The sandbox is Ubuntu 22.04 only and leaves out some system capabilities, so certain attacks reproduce only partially. No Windows, no macOS, no mobile, no GUI. The labeling and the scenario design are manual, which leaves room for bias. On the other hand, every experiment is sealed off from real networks and real secrets, and the scenarios are documented well enough to repeat.