Coding agents solve just 41% of tasks in a new refactoring benchmark

When the old benchmarks get too easy

Benchmarks for coding agents follow a familiar arc. First they push the field forward. Then models catch up fast, scores climb, and the metric stops telling systems apart honestly. At the SWE-bench level you can already see it: frontier models pass those tasks more and more often, and some of the unsolved examples turn out to be broken by bad tests.

Against that backdrop, the authors of SWE-Bench ProMax propose a different way to measure progress. Instead of the usual bug fixes, large refactors in real repositories. Changes where the job is not to invent one patch in one file, but to thread new logic carefully through a project without breaking anything along the way.

And the picture shifts sharply. The best score on the new benchmark is only 41.2%. For today's agents that is no longer comfortable territory.

🟠 SWE-Bench ProMax tests long refactors, not local fixes

🟠 best score — 41.2%

🟠 on the older benchmarks frontier systems had already climbed far higher

What SWE-Bench ProMax is

SWE-Bench ProMax is a benchmark of 170 refactoring tasks taken from real GitHub commits. It covers 7 languages: Python, Java, TypeScript, Go, C, C++ and Rust. Inside are not toy examples but changes from live projects, where edits have to be coordinated across several places at once.

The average task here is noticeably bigger than usual:

🟠 on average 11.4 source files have to change

🟠 the average edit is 261.6 lines of code

🟠 counting tests, that comes to 15.9 files per task

🟠 almost a third of the tasks require edits in more than 10 files

🟠 almost a third of the tasks change more than 200 lines

This matters because refactoring in real development rarely stays inside one file. You change an interface, move types, rename fields, pull shared logic out — and behind that come the call sites, the tests, the configs, the documentation, the build glue. An agent needs more than local care; it needs long-range planning.

The selection and manual-curation pipeline: from tens of thousands of commits down to 170 vetted examples.

Why the old numbers are hard to trust

The authors start with an uncomfortable fact. The trouble with the old benchmarks is not only that they have become too easy. It is also the quality of the grading.

Two distortions showed up often in earlier task sets:

🟣 tests too narrow, rejecting a correct solution when it is not built the way the author expected

🟣 tests too broad, demanding behavior the task statement never described

If you compare agents on tasks like that, the metric starts picking up noise. A model can fail not because it cannot program but because the tests check things the task never asked for. Or the reverse: pass by accident, having guessed the expected shape of the solution.

There is one more problem: data contamination. If the gold patch already appeared in the training data, high scores stop meaning real engineering ability. The model may simply be reproducing a familiar chunk.

SWE-Bench ProMax was built as an answer to both at once: the saturation of the old benchmarks and the doubts about how they grade.

🟠 the problem with the old sets is not just that they are easy, it is noisy grading

🟠 narrow tests can throw out correct solutions

🟠 broad tests can demand more than the task states

🟠 data leakage distorts the results

How the tasks were assembled

The most interesting part here is not only the tasks themselves but the data pipeline.

The authors started from nearly 29,782 candidates and kept only 170. The attrition is enormous. It says a lot about how hard it is to build a benchmark where the task is difficult, the environment reproducible, and the tests honest.

The pipeline had three stages.

🟠 First, the GitHub API was used to find suitable repositories: at least 500 stars, an open license, and one of the seven target languages as the project's primary language.

🟠 Then they pulled commits from after January 2025 whose message contains the word refactor but not bug fix, and which touched both test files and ordinary ones.

🟠 Next they built an isolated environment in Docker, rolled the repository back to its state before the refactor, applied the gold patch and checked that the thing actually runs and passes its tests.

The automated part did not end there. After it came manual work by experts.

The authors rewrote the task descriptions from scratch. Not patching up the commit message, but turning it into a clear specification: what has to change, where the transformation is expected, which properties of the behavior must be preserved. In parallel they read through the tests by hand and threw out checks that were too narrow or too broad.

The description has to be precise enough that a correct solution is unambiguous, and the tests have to check exactly that.

Why refactoring makes a good stress test

A bug fix can often be closed out locally. Find the source of the error, fix the condition, run the tests. Refactoring works differently. It almost always requires holding the structure of the project in your head.

If you change an API interface, one module is not enough. You have to walk every call site. If you move types or extract a shared layer, you cannot forget the build, the imports, the templates, the adapters, sometimes even the code generators. And through all of it the external behavior has to stay the same.

The paper has an example from NASA's F’Prime in C++: a single refactor touched 244 files. That is no longer a "write this function" task. It is a "do not lose the project along the way" task.

Cases like that are much better at showing where coding agents still hit a ceiling.

What happened when the models were run

The authors ran 6 models across two agent frameworks: the lightweight mini-swe-agent and the richer OpenHands. The budget was the same in every case: up to 300 steps and up to $10 per task.

The headline result is blunt: nobody looks confident.

🟣 GPT-5.2 — 41,2%

🟣 Claude Sonnet 4.6 — 38,8%

🟣 GLM-5 — 36,5%

🟣 Qwen3.5 — 36,5%

🟣 Kimi-K2.5 — 32,9%

🟣 Gemini-3-Pro — 19.4% in OpenHands

For context: on the older SWE-bench Verified, frontier agents were already clearing 75%. Here the difference is nearly twofold.

The new benchmark really does measure what the old sets had all but stopped distinguishing: the ability to carry a long, many-part task through to the end.

🟠 best score — 41.2%

🟠 on SWE-bench Verified the scores were already above 75%

🟠 long refactors stay hard for almost every model

Open models have nearly caught up with closed ones

Another interesting part is the economics.

GLM-5 posted 36.5% at an average cost of $0.24 per task. For comparison, Claude Sonnet 4.6 reached 38.8% but cost $4.77 on average. GPT-5.2 — 41.2% at $3.60.

There is a difference in quality, but it is nowhere near proportional to the price.

In short:

🟠 the best score does not mean the best value for the money

🟠 open models have already pulled close to closed ones on hard tasks

🟠 an expensive agent easily burns steps on useless loops without getting closer to a solution

For teams building their own coding agents, that is a practical conclusion. If you are automating internal development, what matters is not only the share of tasks solved but the cost of a single attempt.

The agent framework matters almost as much as the model

There is another layer to this. The same model brain behaves very differently in different agent frameworks.

Almost every model gained noticeably when moving from mini-swe-agent to OpenHands. GPT-5.2, for instance, went from 21.8% to 41.2%. Close to a doubling.

The conclusion is simple: on big tasks the outcome depends not only on the model but on the tools you hand it. Comfortable file editing, command execution, working with the environment, the shape of the look-think-edit loop — all of it feeds straight into the result.

Put differently, comparing bare models apart from their agent scaffolding is no longer very useful. On long tasks the agent framework is part of the system, not an accessory bolted on the outside.

🟠 GPT-5.2: 21.8% → 41.2% on a change of framework

🟠 quality depends on the agent's tools, not the model alone

🟠 the agent framework has to be counted as part of the system

Where exactly agents fail

The authors looked at the agents' trajectories and found a recurring pattern: when a task is not solved, the agent usually changes too few files compared with what the gold patch required.

The problem is not that the agent missed the point entirely. Often it finds the central spot and starts the right refactor, but never carries it out to every dependent piece of the project.

It looks like this:

🟣 the core logic is changed correctly

🟣 some call sites or peripheral modules are left on the old version

🟣 the tests keep failing because the project is now inconsistent

🟣 the agent spends further steps reading, patching in part and reverting, but never widens its coverage

That is why the authors name the main failure mode incomplete refactoring.

This lines up well with practice. The task is unpleasant for a human too: you change an interface in one place, then the old call sites surface, then the test fixtures, then a config, then the documentation, then some rare execution path. You have to keep the map of changes in your head. Agents visibly struggle with that so far.

The languages do not behave the way you would expect

The benchmark is multilingual, and that matters too. Evaluations of coding agents are usually tied far too tightly to Python. Here you can see how models behave in different ecosystems.

The picture came out uneven:

🟠 GPT-5.2 is best of all on Python and C

🟠 Claude Sonnet 4.6 leads on TypeScript and Rust

🟠 GLM-5 beats the rest on Java

🟠 Qwen3.5 posts the best score on C++

🟠 Kimi-K2.5 looks stronger on Go

Curiously, TypeScript and Rust did not turn out to be equally hard for everyone. Some models handle them well, others fall apart on them. So it is not only the intrinsic difficulty of the language, but also the data and the code patterns the model learned from.

Co-occurrence matrix of task types: refactoring often travels together with API changes and bug fixes.

Refactoring is almost never clean

One more point: the tasks in SWE-Bench ProMax rarely reduce to refactoring in the narrow sense.

By the authors' own labeling, many of the examples include several of these at once:

🟣 a change to the API interface

🟣 tidying up the structure of the code

🟣 a bug fix

🟣 adding a new feature

🟣 updating tests or documentation

That looks a lot like real development. In practice you rarely see a commit where the author renamed a field and did nothing else. As soon as you touch the architecture, old defects tend to surface, or neighboring parts of the system have to be brought along.

For an agent that means one unpleasant thing: mechanically repeating a pattern file by file is not enough. You have to understand where the refactor drags semantic changes along behind it.

Why this matters for the whole field

If you want to measure the progress of coding agents honestly, short tasks are no longer enough. They are still useful, but they do not answer the main question: can an agent finish work at the scale of a project.

SWE-Bench ProMax shows three things at once.

🟠 today's agents are much better at local edits than at long-range coordination across a codebase

🟠 a benchmark's quality depends not only on its size but on the manual review of its descriptions and tests

🟠 open models can already compete with closed ones where the gap used to look wider

And, probably the most practical point: the bottleneck right now is not abstract reasoning in general, but the ability to hold a plan of changes across many files and see it through without it decaying into partial patches.

The bottom line

SWE-Bench ProMax moves the evaluation of coding agents closer to real engineering work. Instead of convenient one- or two-file tasks, it has many-part refactors where you have to change interfaces, keep dependencies in sync and preserve the behavior of the project.

The results sober up expectations quickly: even the best models solve about two fifths of the tasks. The core problem is not that the agent fails to find the idea behind the edit, but that it does not propagate that idea to all the code that needs it. Hence the frequent failures: the central part is done, the periphery forgotten.

The next step for coding agents takes more than stronger models. It takes better agent frameworks, more reliable handling of repository context, memory of the changes already made, and mechanics that keep coverage from slipping. Otherwise, on large refactors the agent will keep running into the same limit again and again: it understood what to do, but never carried it all the way through.