AI coding agents still need manual rework on most real-world tasks

Researchers from Alibaba and several Asian universities created Co-Design Bench, a benchmark for testing AI agents on real requests to modify web applications. Ten models took part; the top performer, GPT-6 Astra, completed about 70% of requirements on average but fully solved just 26.4% of tasks. An agent usually does most of the work, but almost always leaves something unfinished that a person has to fix manually.

Most AI coding tasks still need manual rework

Testing with real requests

Co-Design Bench tests how agents modify existing web applications in response to user requests. Inputs can include images, audio recordings, video and documents.

The authors argue that familiar benchmarks have become too easy: models that can run, test and fix their own code score above 99% on them, making it hard to distinguish between systems. Instead, the researchers collected requests from real coding sessions. They selected cases where users had to rephrase a task several times or undo changes, so the test includes tasks that had already challenged AI agents in practice. The range runs from small edits, such as extracting a total from a receipt and updating all related figures, to a complete project overhaul.

The evaluation is also stringent. One AI judge checks the finished application in a browser, while a second reviews every positive result and looks for mockups and hard-coded answers. Credit is given only for features that actually work in the application.

On older benchmarks like WebCompass-edit, agentic coding models reach near-perfect scores, while Co-Design Bench leaves a gap of more than 35 percentage points. | Image: Wei et al.

Source: the-decoder.com

Complex changes make tasks harder to complete

After GPT-6 Astra, the rankings list Claude Opus 5 and Qwen3.8-Max, with Grok 4.6 in last place. On major overhauls, the smaller Qwen3.8-27B rises to second place, but the authors do not consider this evidence of a general model advantage.

All models perform worse when a task requires changing the layout, design, interactions and content at the same time. Such combinations of requirements are common in real projects.

More compute does not solve the problem. On smaller changes, GLM-5.2 uses almost six times as many tokens as GPT-6 Astra, yet still trails it by a substantial margin.

In the multimodal track, agents need to extract values from receipts, tables, or videos and update all dependent views consistently. | Image: Wei et al.

Source: the-decoder.com

The execution environment affects results

In a separate experiment, the researchers ran the same models in different environments — programs through which an agent reads, changes and runs code. On major overhauls, Qwen3.8-27B scored 75% in one environment and just 56% in another. The environment also determined which of the two models performed better.

Every one of the ten models scores worse once a task demands three or four different types of changes. | Image: Wei et al.

Source: the-decoder.com

Rankings that list only the model therefore give an incomplete picture. The authors recommend reporting both the model and the environment it runs in. Companies choosing a coding agent should evaluate these components together.

Co-Design Bench has limitations: each task comes with one fixed request, and the benchmark does not test follow-up questions or user corrections during the process. AI judges can also make mistakes.

METR’s research has already shown a gap between benchmark results and the practical usability of code: project managers accepted AI-generated code about 24 percentage points less often than the results of the popular SWE-bench Verified suggested.

Weaknesses in the test environment may also affect scores: because of them, an agent can sometimes pass a task without actually solving it. At the same time, the environment’s role is not entirely clear-cut. In another recent study of coding agents, more complex environments yielded almost no measurable improvement, despite using up to three times as many tokens.

GPT-6 Astra leads both tracks but fully solves only 41 percent of multimodal tasks and just under 18 percent of repository tasks. | Image: Wei et al.

Source: the-decoder.com

Dataist’s take: Co-Design Bench shows how much agents’ results depend on the complexity of real requests and the conditions in which they run. Its scores are best considered alongside the test’s limitations: users cannot clarify a task as work progresses, and AI judges conduct the evaluation.