A multi-agent pentester outscored 9 of 10 humans on a live network

The argument over how good AI agents are at cybersecurity work has been running hot for a while now. It usually rests on one task: finding known vulnerabilities. But real penetration testing doesn't look like that. It's a large corporate network, thousands of hosts, incomplete (and more often unreliable) data, chains of small vulnerabilities and the human factor. Which is precisely where it matters that you can do more than “know the answer” — you have to plan the investigation and carry it through to the end.

Researchers at Stanford set out to run exactly that comparison: real cybersecurity professionals against AI agents, inside a live university network of roughly 8,000 hosts across 12 subnets.

That is the main virtue of the work. What gets measured here isn't abstract cyber trivia but the ability to operate inside real production infrastructure.

Why this is hard and risky

Unlike a sandbox exercise, work against production carries operational risk: aggressive vulnerability scanning can hit the availability of the infrastructure; finding and exploiting a bug can affect data integrity; and handing artifacts to the wrong people affects confidentiality.

So the authors agreed the whole process in advance, in a “real pentest, but safely” mode: destructive actions were forbidden, the agent's range of action was bounded, and the entire IT team watched it work. That is an important part of the experiment.

A pentest isn't only about how many holes the agent finds. It's about how hard the hole is technically and how critical it is to the business. So the authors built a scoring scheme that accounts for both technical difficulty and business impact. They also record separately whether a vulnerability was actually exploited or only confirmed without full impact — the second earns fewer points.

The researchers hired ten people and told them to work at least 10 hours each and to document the vulnerabilities they found in a supplied template.

The AI side was set up more interestingly: six existing agents plus ARTEMIS. ARTEMIS has a supervisor, some number of subagents and dynamic prompt generation. Incidentally, the authors point out that ARTEMIS adds no new knowledge to the LLM at all.

What happened in practice

The human group turned up 49 unique vulnerabilities. The spread across individuals is wide: one person brought in 3, another 13. A given vulnerability was usually found by one or two participants.

With the AI agents the picture is the opposite. Many of the existing agents ran out of steam quickly, or got stuck at the opening stage, or flatly refused to work out of the box because of their internal guardrails. ARTEMIS did better: it kept working for long stretches, parallelizing tasks and returning to its original objectives.

On the final leaderboard, one ARTEMIS configuration came second: 11 vulnerabilities found, 82% of submissions judged valid, a final score of 95.2. That beats 9 of the 10 humans. But another ARTEMIS build produced far worse precision (55% valid), which illustrates the paper's central point perfectly: the result comes from the scaffolding as much as the model. The same engine in different harnesses behaves differently — sometimes the outcomes diverge completely.

Where the AI agent is strong and where it is weak

ARTEMIS does especially well where the job calls for method, persistence, parallel attempts and memory across long scenarios. It's rather like several juniors working under a sharp tech lead who never gets tired and forgets nothing. There is an economic effect as well: by some estimates ARTEMIS ran at $18/hour against roughly $60/hour for professional specialists.

Where it loses on quality is in synthesizing signals. Agents have a higher false positive rate, and that is no small thing: every spurious alarm on the defensive side costs engineer time and risks misprioritization. Another real limit today is anything that needs a graphical interface. Where a pentester simply clicks something quickly to check a hypothesis, the agent can spend minutes, or even hours, backtracking.

What this means for us

The study shifts the focus away from whether agents can break into IT systems, toward a more practical question: which methods and tools turn an AI into a dependable pentester?

The conclusion isn't sensational, but it is worth having: ordinary agents still trail humans at finding vulnerabilities, but a well-designed multi-agent system can compete with strong specialists — and in places outrun them on process consistency and the scale of parallel work.

Security, though, also demands control over the agents themselves: inspection of their reasoning chains and a carefully considered access policy for the systems they touch. Without that, agents are capable of doing something very dangerous.

💾 Code