Real customers are stumping AI agents.

When the Code Works but the Business Doesn’t

An AI agent can do exactly what a customer asks, pass every test, and still fail to improve the way the company works. Sometimes that’s because the customer asked it to fix the wrong problem.

This article describes a case from the drilling industry. A manager wanted daily summaries and a chat interface for asking questions about wells. But the real losses were happening elsewhere: the system was incorrectly assigning downtime to a contractor, costing the company its right to compensation. Fixing the problem meant understanding the contract, talking to employees, and changing code the customer had explicitly ruled out of scope.

That’s the work of an engineer embedded with the customer’s team: not just writing code to match a brief, but figuring out what actually needs to be done. The authors introduced the FDE-Gym benchmark to test whether today’s AI agents can handle this kind of work.

What Is FDE-Gym?

Traditional benchmarks often give an agent a clearly defined task: fix a bug, answer a query, or build an app from a specification. In real projects, requirements may be incomplete, documents out of date, and the customer’s request may point in the wrong direction.

FDE-Gym models the full delivery process. An agent examines company files and data, communicates with employees, modifies a live system, and then has its work evaluated on production data held back from the agent.

The benchmark includes 95 tasks across 20 scenarios and 18 industries, from banking and healthcare to energy and robotics. Specialists created the tasks based on real projects. Variants within each scenario change important facts—for example, who has the information the agent needs, or which rule is outdated. So an agent can’t pass simply by memorizing a standard answer.

How real projects are turned into FDE-Gym tasks, complete with a working environment, altered facts, and a way to evaluate the results.

Agents work in a copy of the customer’s system. They can read documents, run code, and message employees. But communication comes at a cost: messages, test runs, and waiting time all count. Excessive outreach or violations of the rules lower the score.

The score reflects business impact. A score of zero means the system is no better than it was to begin with. A score of one means the result matches an expert-built solution. If the agent makes the system worse, its score goes below zero. Breaking the stated rules also rules out a positive score.

Tasks vary in how much information the agent receives. At the first level, the problem, rules, and success criteria are spelled out. At the last, the agent gets only the customer’s initial request—which may be aimed at the wrong problem. The environment and evaluation method stay the same.

Model results at different levels of task information, and across all tasks in the most realistic setting.

Success Means Improving the Customer’s Business

The authors tested 13 model configurations from seven providers. All used the same simple harness and had access to the same tools. Each task was run three times. In the most realistic setting, where the agent had to work out the real problem and uncover hidden constraints, Claude Opus 5.5 came out on top, with an average score of 0.56 out of 1.

But that average hides the bigger picture. The model solved just 14 of the 95 tasks—meaning it scored at least 0.8 without breaking any rules. Even if you pick the best result across all 13 configurations for each task, the average rises only to 0.70, and the number of solved tasks reaches just 30.

• Best model’s average score: 0.56 out of 1
• Tasks solved by the best model: 14 of 95
• Tasks solved by selecting the best model for each task: 30 of 95
• Configurations that solved no tasks: 5 of 13

Models often found most of the information they needed, but that didn’t guarantee a good solution. On average, the three leading models found 72% to 85% of the important facts. Yet their solutions delivered only 60% to 70% of the improvement achieved by experts.

Even when Claude Opus 5.5 uncovered every key fact, it achieved 80% of the expert result in only 10 of 23 cases. Gathering information isn’t enough. The agent also has to use it well: choose the right solution, integrate it into the existing system, and respect operational constraints.

How the facts an agent uncovers relate to business impact, along with the cost of using each model.

Conversations with employees also affected the final score. For example, Claude Opus 5.5 improved business outcomes by 0.70 on average, but the cost of its outreach knocked off 0.14 points. For GPT-6 Astra, those costs reduced the score by 0.22. The models weren’t told in advance how much each message would cost. They had to work out for themselves who to talk to and when.

The Hardest Part Is Finding Hidden Constraints

The authors compared results across four levels of task detail. When all the rules and constraints were provided upfront, the top three models scored between 0.73 and 0.82 on the 20 original tasks. When the problem was still clearly described but the rules had to be uncovered, their scores fell to between 0.39 and 0.62.

The biggest challenge is discovering what the customer hasn’t mentioned. That means searching documents for relevant details, comparing them with how the system behaves, and asking the right people the right questions.

When the request is vague or misses the real problem, the top models’ results decline less than the authors expected. One possible explanation is that, as the model investigates the constraints, it also gets a better sense of what the customer actually wants. But even a full description of the rules doesn’t make the engineering challenge go away: with the most detailed task descriptions, no model scored higher than 0.82.

Results also vary widely by scenario. In five of the 20 scenarios, no model averaged even 0.5. In the two hardest scenarios, the best scores were 0.18 and 0.27. And changing the facts in different versions of the same task could shift the score substantially.

Model scores by scenario: results vary by industry and by the specific facts of the task.

There’s also a risk of making things worse. Several weaker models often scored below zero, meaning the customer would have been better off leaving the system alone. Claude Haiku 4.5, for example, scored below the starting point on 79 of the 95 tasks. If a system misallocates payments or changes the wrong process, a successful deployment isn’t a success.

Why Traditional Benchmarks Don’t Tell the Whole Story

A model can perform well on coding or information-retrieval tasks and still struggle to deliver systems for customers. These are different kinds of work. Code tests show whether an agent can implement a specified solution. FDE-Gym tests something else, too: whether it understood which solution was needed, stayed within the rules, and improved the company’s operations.

The authors compare their results with several well-known benchmarks for coding, app development, and financial research. The numbers aren’t directly comparable: those benchmarks typically count the share of tasks completed, while FDE-Gym measures normalized improvement in a business metric. But the scale of the gap is clear: models that solve a large share of tasks on those benchmarks improve the business metric on at most 14 of 95 FDE-Gym tasks.

The benchmark also has limitations. Customer employees respond using prepared scripts, while real conversations are less predictable. Results also depend on the harness used, and human specialists haven’t yet been tested in the same way, so there’s no direct comparison between agents and engineers.

Conclusion

For a real customer, it’s not enough for an AI agent to write working code or carry out the initial request. It has to understand what’s actually getting in the company’s way, uncover hidden constraints, and deliver measurable improvement without unnecessary risk or cost.

FDE-Gym highlights the gap between solving a clearly defined task and delivering a working system that benefits the customer. Even the best model currently solves about one task in seven. The ability to ask questions, gather facts, and write code doesn’t guarantee that the final solution will help the business.