Top coding agents solve under a quarter of SWE-Bench Pro tasks

Over the past couple of years, agents built on large language models have settled into everyday development: they read repositories, fix bugs, propose patches and run tests. On the classic SWE-Bench Verified, the top systems clear more than 70% of tasks on the first attempt. The trouble is that progress like this creates a false sense of readiness for real industry work. Out there, tasks rarely fit on a single line, and they almost always demand changes in several places, interfaces that agree with each other and careful handling of dependencies. Researchers have put together a harsher proving ground — SWE-Bench Pro — and the picture is sobering: even the strongest models are still far from the level of a professional engineer. For now, you can sleep soundly.

What they assembled

SWE-Bench Pro holds 1,865 tasks from 41 actively maintained repositories: business applications, B2B services and developer tools. The set is split three ways — a public open portion (11 repositories), a held-out portion (12) to guard against overfitting, and a commercial portion (18) from private codebases, where only the results are published.

The main difference is difficulty. One- and two-line micro-edits are out: every task requires at least 10 lines of changes. The average reference solution is 107 lines of code across 4 files, and more than a hundred tasks cross the 100-line mark. That scale is routine inside companies, where editing one function almost always drags interface and test updates along with it.

How the tasks were built

The starting point is a pair of consecutive commits: a base commit and the one that fixes a bug or adds a feature. The tests are split into a verification patch and the working solution. With the new tests applied, the base version has to fail; with the reference changes, it has to pass. This is the classic fail2pass setup.

To strip out ambiguity, the authors added to every task by hand: a problem description, a list of requirements and, where needed, an explicit interface — class names, method signatures, expected routes. The runtime environments are packaged as Docker images for Python, Node.js and Go, and every instance was screened for flaky tests through repeated runs and manual review.

How the evaluation ran

The team used a single harness based on SWE-agent: up to 200 steps, tool access, one unified prompt. The headline metric is Pass@1 — the share of tasks solved on the first attempt with every test passing. A model-as-judge pass was added on top to classify the typical failures: algorithmic errors, syntax, mistakes in tool use, context overflow and so on.

The results

On the public dataset, the best result belongs to GPT-5 at 23.3%. Claude Opus 4.1 scored 22.7%, Sonnet 4 17.6%, Gemini 2.5 Pro Preview 13.5%. Among open models, SWE-Smith-32B managed 6.8%, GPT-4o 4.9% and Qwen-3 32B 3.4%. On the commercial dataset the numbers drop further: 17.8% for Opus 4.1, 14.9% for GPT-5, single digits below that. It is a sharp contrast with earlier benchmarks, where strong systems closed the lion's share of the tasks.

By language, the picture is uneven. Models are noticeably more assured on Python and Go, while JS/TS produces lower and shakier accuracy. The spread across repositories is wide as well: there are projects where solutions almost never pass, and there are visible pockets of higher success.

Where agents go wrong most often

The failure analysis shows that the larger models tend to miss on meaning — logic and algorithms on big edits — while the smaller ones miss on syntax and tooling. Some systems ran into context overflow and endless walking of the file tree, formatting errors and build incompatibilities. Put another way, a long development horizon exposes not just how well a model understands the task but whether it has engineering discipline: making consistent changes in several places, bringing up an environment, fixing migrations and test fixtures.

Why this matters

SWE-Bench Pro pulls the evaluation of AI agents back to the scenarios companies actually need them for: multi-file patches, substantial interface changes, tangled dependencies and test suites close to production grade. The visible slump to roughly 23% is not a failure but an honest baseline. A benchmark like this measures progress where automation is worth the most, and it hands over diagnostic signals: where context runs out, where the tooling falls short, where a better world model of code is needed.

What comes next

The authors plan to widen language coverage (Java, C#, Rust, Kotlin), to move past "tests passed / tests failed" toward metrics for quality, security and performance, and to add collaborative development scenarios: multi-agent systems, code review, merge conflicts, distributed pipelines. The direction of travel is evaluating engineering work as a whole.

💾 Code