Four agent roles and a real PR review get 72.4% on SWE-bench

LLMs can suggest a chunk of code, explain an error, or write a test. But the moment a task starts to look like real development work — read an issue, find your way around a project, reproduce a bug, produce a patch without breaking everything else — a single general-purpose agent often falls short. In Agyn: A Multi-Agent System for Team-Based Autonomous Software Engineering the authors put forward a simple idea: maybe the problem isn't only model size, but that we make the AI work in a shape that doesn't fit software development.

In ordinary engineering practice, one person rarely does everything alone, from investigation through review. There are roles, communication, checks on decisions, iteration, and above all a workflow everyone understands. The authors carry that logic over into autonomous development and build a multi-agent system that imitates a team.

A process, not a monolith

The central goal of the work is to make autonomous issue-closing look more like what happens in live teams. Not "generate a patch", but walk the whole path: understand the problem, state a plan, implement it, open a pull request, run the review, address the comments, and finish only after approval.

To do this the authors build on top of the agyn platform (open source), where a team of agents can be assembled as an "organization". It is not a pipeline with fixed steps, and not one super-agent, but a set of roles, each with its own responsibility and context. Crucially, the agents work in isolated sandbox environments: each one can experiment, run tests and try alternatives without breaking anyone else's work. It resembles the way developers keep local branches and environments while the shared result converges through the PR.

The roles on the team, and why this works at all

The lineup the authors chose is fairly human: a manager, a researcher, an engineer and a reviewer.

The manager holds the whole picture: deciding who to bring in next, and when to stop investigating and move to edits. The researcher digs into the issue and the codebase and produces a specification — what exactly needs to change, and where to look for the cause. The engineer edits code, runs tests, debugs. The reviewer reads the diff in the PR and leaves line-by-line comments until it is ready to say "approved".

One interesting detail: the roles can run on different models. The more deliberate models go to analysis and planning, while cheaper, more specialized ones handle implementation and debugging. That mirrors production logic: it isn't always worth paying for an expensive model where iteration speed matters more.

GitHub as the single task board

The authors deliberately make the process GitHub-native: issue → pull request → inline review → fixes. And it is not a simulation inside a chat window but real work with GitHub artifacts. For interaction they ended up preferring the gh CLI over the API, because the API bloats the context. For inline review they even had to write an extension, because stock gh is not much good in that role.

There is one more practical touch: the system keeps command output in check. If a log is too large it goes to a file, and the agent reads the parts it needs later. This looks like a small thing, but on long tasks it is exactly this kind of detail that saves you from a ballooning context and degrading quality.

The SWE-bench check, and an unexpected emphasis

Although the system was not built for a benchmark, the authors evaluated it after the fact on SWE-bench 500 — the popular set of real GitHub issues where a task counts as solved only if the patch passes the project's tests. In fully autonomous mode, with no human involved, their approach solves 72.4% of the tasks. That is competitive against strong baselines in the same 70–75% range.

But in the discussion the authors are honest about where the process breaks. SWE-bench holds a lot of repositories with aging infrastructure: dependencies are out of date, and CI can fail over deprecated GitHub Actions. In production a team can usually spend the time to restore the environment — on a benchmark it turns into a trap, and the agent wanders off fixing everything around it instead of fixing the specific bug. Hence an important conclusion: even a good workflow can lose if the ground underneath it is unstable.

What ultimately matters in this work

The paper's main value is the demonstration that progress in autonomous development is bounded not only by better LLMs, but by organizational design: roles, communication rules, completion criteria, environment infrastructure and the handling of artifacts. Here "knowing how to work as a team" becomes an engineering multiplier in its own right.

The authors do not claim to have found the perfect scheme. But they show a realistic path: less magic, more process discipline. And that is probably one of the most practical ways to bring autonomous systems closer to real development, where what counts is not only the right answer but reproducibility, review, traceability and careful changes.

💾 Code