Coding agents score 21% when asked to evolve a codebase between releases

Coding agents have gotten noticeably better over the past year: they can find where something broke, edit the code and run the tests. There is an important caveat, though. Most popular benchmarks measure point achievements — fixing one specific bug, or adding a small feature scoped to a single issue. Real development does not work that way. Code lives for years, requirements shift, dependencies ship new versions, and a change has to be spread across dozens of files without breaking anything on the way.

The authors of SWE-EVO propose facing facts and measuring something closer to everyday engineering: an agent's ability to evolve a codebase from one release to the next, working not from a neatly written issue but from release notes — short bullet points about what changed and what is supposed to work in the next version.

Why the old checks are no longer enough

The authors' reasoning is simple: over a project's long life, what matters is not only "the right lines of code" but the ability to act like a developer who reads high-level requirements, plans the changes, works across many modules and keeps compatibility intact. In tasks like that it is easy to make the right edit in the right place and still break the logic in a neighboring subsystem — and the tests will show it.

SWE-Bench and similar benchmarks brought evaluation closer to reality, because they are built on real GitHub issues and check the patch with tests. But those are still mostly single, relatively isolated tasks. SWE-EVO is built specifically so that the agent has to cover the long distance from the text describing a release's changes to a sequence of edits across the entire repository.

How SWE-EVO is built

SWE-EVO is assembled from seven mature open-source Python projects. It holds 48 tasks in total, each one a move from one version to the next. The problem statement handed to the agent comes from the release notes (essentially a mini-SRS at the product level), and grading is done by running tests in a pinned environment.

On average a change touches about 21 files, and a single task carries roughly 874 tests. In some cases there are thousands. So the agent does not just have to generate a patch — it has to hold a large context and avoid causing a regression.

The authors make a point of compatibility with existing agent frameworks: the benchmark is "plug-and-play" for SWE-Bench infrastructure, so it is easy to pick up and run on a typical setup.

What counts as success

As in SWE-Bench, the headline metric is the share of fully solved tasks: every test that was supposed to stop failing passes, and nothing that worked before breaks. But in long-horizon scenarios that binary becomes too blunt. With hundreds of failing tests, an agent can fix half of them and still score zero because of what is left.

So SWE-EVO introduces Fix Rate: the share of failing tests the agent managed to fix. That gives a fairer read on partial progress — useful both for analysis and for comparing approaches while the absolute percentages are still low.

What the experiments show

The authors ran 11 current LLMs inside two agent scaffolds — OpenHands and SWE-agent — and tested two ways of stating the task: the release note alone, or the release note plus context from linked PRs and issues, where such links exist.

The picture that emerges is fairly harsh. The best result comes from GPT-5 paired with an agent: about 21% of SWE-EVO tasks solved. For comparison, the same systems land around 65% on SWE-Bench Verified. Adding PR/issue context helps, but only moderately: extra text by itself does not make the task easy, because the hard part is holding on to the meaning of the changes and distributing them through the code.

The breakdown of failures is interesting too. The large models have almost no problems other than understanding the requirements: the agent read the release note but implemented the wrong thing, or missed an important detail. Weaker models add tool-calling errors, loops and early exits.

What this changes for coding agents

SWE-EVO is not an attempt to bury existing evaluation methods, more a return to reality. The benchmark shows that agents can fix bugs, while evolving a system over the long run remains hard. And the reason has less to do with tool calls than with semantics: you have to read the spec correctly, build a plan of changes, carry out the refactor cleanly and prove it with tests.

It is also good to see the authors trying to measure progress more honestly by introducing Fix Rate: in tasks like these, it matters to see more than a binary win or loss.

💾 Code