An agent that debugs its own scaffolding gains 9 to 10 points on three benchmarks

An AI agent can fail a task over one small thing: it didn't check a file path, it missed the second matching trip, or it decided too early that the work was done. In a short exchange this looks like a stray error. In a long task, that error drags the following steps along with it — and the whole result collapses.

The authors of AutoSaddler propose repairing such failures automatically. They change the language model's scaffolding itself: the system prompt, the tools, the control logic and the intermediate checks. The result is a kind of debugging loop in which the AI agent analyzes its own failures, edits the scaffolding code and checks whether anything got worse.

By the authors' numbers, this raised success rates on three hard benchmarks by 8.4–10 percentage points over the original systems.

Why the prompt is no longer enough

Modern AI agents can call tools, read files, work in a terminal and carry out chains of actions. But their abilities stay uneven. An agent can solve one task cleanly and fail an almost identical one — because of different wording, a different order of steps, or the state of the environment.

In long tasks the problem compounds:

🟠 The agent misread one parameter and kept working with the wrong data.

🟠 A tool returned an incomplete result, and the agent missed the limit.

🟠 The system prompt gave a rule so general that it fired in the right context and the wrong one alike.

🟠 Fixing one error accidentally broke scenarios that used to work.

Normally a developer reads the execution log, guesses at the cause and edits the prompt or the code by hand. For long trajectories that is expensive. You have to work through dozens of tool calls, find where exactly it broke, and then test the new version across many tasks.

AutoSaddler turns that work into training the scaffolding without updating the model's weights. The model does not update its weights. Instead, the system gradually changes the external components, using accumulated execution trajectories and task outcomes.

What AutoSaddler is

AutoSaddler has three specialized AI components:

🟣 The diagnosis and repair component reads failed trajectories, studies the scaffolding code and looks for the root cause of the error.

🟣 The reflection component compares behavior before and after a fix. It records which scenarios were repaired, which got worse and which stayed the same.

🟣 The evolution component picks the base for the next version. It can continue the current branch, bring back a useful change from an earlier candidate, or merge fixes from different branches.

Instead of a plain list of changes, the system keeps its history in a directed acyclic graph called EvoDAG. Its nodes hold scaffolding versions, scores, descriptions of fixes and conclusions about what worked. The edges show which set of changes produced the next candidate.

The AutoSaddler loop: run on a minibatch of tasks, diagnose, fix, re-run, measure generalization and pick the next scaffolding version.

A single iteration looks like this:

🟠 The current scaffolding runs on a small minibatch of tasks.

🟠 The system collects the outcomes and the full trajectories: tool calls, responses, reasoning and environment state.

🟠 For the failed scenarios it forms a hypothesis about the root cause.

🟠 A structured fix is generated. It can change a prompt, a tool or the control logic.

🟠 The new version runs again on the same minibatch.

🟠 If the result improved, the candidate is checked against a held-out dev set.

🟠 The outcomes go into the EvoDAG and feed the next iteration.

The authors compare the process to minibatch training. In place of a numeric gradient there is the chain diagnose → fix → verify. It plays the role of a textual update direction: the system has to do more than notice an error, it has to propose a testable change.

Which parts of the scaffolding change

AutoSaddler does not let the agent edit whatever it likes. The space of changes is split into three layers:

🟣 Prompt — the behavior rules and instructions for the model.

🟣 Tools — new calls, parameters, descriptions and function implementations.

🟣 Middleware — pre-tool-call hooks, the agent loop, environment setup and runtime control.

There is a second split as well. Capability fixes change what the agent is able to do at all: they add a tool, repair its implementation or the loop logic. Control fixes change how the agent uses what it already has: they sharpen the prompt, a tool description or a reminder issued before a call.

The search runs in two stages. First the system looks for capability fixes. That clears structural limits out of the way. Then it moves on to finer changes in behavior.

Distribution of fixes: left unconstrained, the system almost always edits text instructions, while AutoSaddler explores tools and control logic far more often.

The constraint exists for a practical reason. Give an AI agent full access to the repository and it usually takes the easiest path — add one more rule to the prompt. But no prompt helps if the tool cannot search the data, the scaffolding mishandles a file path, or the agent loop hangs.

In the experiments, capability fixes found roughly as many local solutions as text changes did, but caused fewer side effects: 8% regressions against 17%.

Results

The authors tested AutoSaddler on three task sets:

🟠 GAIA2 simulates an assistant at work inside a smartphone's digital environment.

🟠 SWE-Bench Pro holds long code-fixing tasks in real repositories.

🟠 Terminal-Bench 2.0 tests the agent on the command line: administration, machine learning and security.

To measure generalization, the tasks were split into groups. In SWE-Bench Pro, for instance, training ran on one set of repositories, development on another and testing on a third. In GAIA2 the groups corresponded to different virtual worlds and personas.

Search efficiency on GAIA2: AutoSaddler reaches higher accuracy with fewer task runs.

On the test sets the results look like this:

🟣 On GAIA2 the original system scored 53.0% and AutoSaddler 62.0%. A gain of 9.0 percentage points.

🟣 On SWE-Bench Pro the baseline agent reached 37.3% and the optimized scaffolding 46.9%. A gain of 9.6 percentage points on the authors' headline comparison.

🟣 On Terminal-Bench 2.0 the score rose from 40.0% to 50.0%, that is by 10 percentage points.

On GAIA2, AutoSaddler also beat two automatic methods that rewrite prompts or the scaffolding wholesale. The sharpest difference showed up in search efficiency. The best dev-set result came after roughly 147 trajectories spent on training. Meta-Harness needed about 1,400 — nearly ten times more.

AutoSaddler's per-step cost is higher: it has to run extra diagnosis and analysis processes. But the tasks themselves are checked selectively. The dev set is used only for fixes that already improved the minibatch. The total spend on the expensive runs therefore comes out lower.

Fixes against regressions: checking on a held-out set keeps the net result positive when local changes start hurting other scenarios.

Why deep diagnosis matters

The most telling comparison is with stripped-down versions of the system. One of them dropped the deep trajectory analysis and kept a single model call asking for the cause of the failure. The GAIA2 score fell from 62.0% to 57.8%.

In the appendix the authors walk through several cases.

In a calendar task the agent mishandled the word "event" when a single day held several cancellations. The shallow analysis concluded the model had simply misread the instruction. The deep diagnosis checked the calendar entries themselves and found the ambiguity. A rule then went into the prompt and the hook: when several objects match, ask for clarification.

In a file task the agent created a directory in the filesystem root rather than inside `Documents/benchmarks`. The shallow analysis blamed a wrong relative path. Reading the logs and the filesystem state showed something else: the agent never checked where the parent directory was. The fix added an automatic lookup of the existing parent path before create and move operations.

In a travel task the agent saw only one matching trip and booked it on the spot. The deep diagnosis went through the entire history, found a second trip on the same day and added a search tool with date and route filters.

These cases show the difference between describing an error and explaining it. The shallow system says: "the agent reasoned incorrectly." AutoSaddler tries to establish which call, file, parameter or environment limit actually produced the error.

Where the system can go wrong

AutoSaddler's main limitation is its dependence on labeled tasks and a legible success criterion. In a benchmark it is easy to tell whether a task was solved. In a real product, the answer a user gets often cannot be scored with a simple right-or-wrong mark.

There are other limits:

⚫ The optimization assumes independent tasks with no long-term state or memory.

⚫ Running the scaffolding itself takes many LLM calls and access to execution logs.

⚫ Two runs can produce different trajectories because of model randomness, so the causal link between a change and an improvement is not always clear.

⚫ Automatic edits to code, prompts and hooks need human review and separate safety testing.

The authors also show that AutoSaddler can carry part of the improvement over to a weaker model: swapping Claude Opus 4.6 for Claude Haiku 4.5, the GAIA2 gain held at 5.6 percentage points over the original scaffolding. But that is one transfer scenario so far, not proof that the method generalizes.

AutoSaddler's search trajectory: the system takes a regression, falls back to a version that worked and merges the fixes that held up.

The takeaway

AutoSaddler offers a practical view of how to improve AI agents. The problem often sits around the language model: in a tool description, the action loop, file handling, the rules for stopping, or a hook drawn too broadly.

A useful process looks like this:

🟣 Start by capturing the full execution trajectory.

🟣 Then find the root cause instead of naming the symptom.

🟣 Then make one small structured change.

🟣 Test it on the original task and on held-out scenarios.

🟣 Keep the conclusion that generalizes, and do not mistake a local win for an improvement to the whole system.

AutoSaddler's results show that an AI agent's scaffolding can be optimized like a software system. For long tasks that may matter as much as the next increase in model size.