Keeping a wiki of failed attempts makes agent skills improve faster

When memory matters more than another prompt

AI agents can already search the web, write code, work with files and handle multi-step tasks. But an old problem remains: they are bad at accumulating experience. An agent fails a task, someone reads the logs, patches a skill, runs it again — and half the useful observations dissolve back into the history of iterations. On the next cycle the system often starts learning almost from scratch.

The WikiSkill paper from Google Research makes a simple proposal: if an agent has learned something from its own mistakes, that knowledge should be stored separately and systematically. Not only as the current skill, but as a persistent knowledge base that collects recurring patterns, the reasons behind failures, and the techniques that work. This wiki layer outlives any single edit to a skill and becomes the foundation for the next round of skill updates.

If you build AI agents, you soon run into a different question: can the system improve itself predictably. That is exactly what WikiSkill is about: how to turn scattered experience from agent runs into a reproducible improvement procedure.

What WikiSkill is

WikiSkill is built on splitting the agent's workspace into three layers.

The WikiSkill design: raw logs, a persistent knowledge base and active skills work as separate layers of one pipeline.

The first layer is raw execution traces. These are complete logs of what the agent did: which steps it took, which tools it called, where it went wrong, where it got things right.

The second layer is the wiki. This is no longer a log but distilled knowledge: common failure modes, strategies that worked, a record of past changes, and the history of which skill edits were accepted or rejected.

The third layer is skills: executable instructions for the agent — what to do in similar situations, which steps to take, what to avoid.

The key idea is that skill and knowledge are kept apart. A skill can be rolled back if it made results worse. The wiki, meanwhile, persists and keeps accumulating experience. So even a failed attempt is not wasted.

The mechanics, briefly:

🟠 The agent solves training tasks with its current skills and leaves logs behind.

🟠 A separate module parses those logs and writes recurring patterns into the wiki: what fails, what helps, which ideas have already been tried.

🟠 A skill proposer reads the wiki, looks at the fresh logs and makes one atomic edit: create a new skill or amend an existing one.

🟠 A validation check decides whether the edit stays or is rolled back.

During training runs the agent itself does not read the wiki. The authors show separately that giving the agent access too early makes it lean on the wiki while solving tasks instead of turning that knowledge into a durable skill. The execution traces then become less useful for further improvement.

Why ordinary skill evolution is no longer enough

Recent work on skill evolution is built around the same loop: run → analyze → edit → validate. But it shares one flaw: knowledge stays smeared across the optimization history. Some of it sits in a rejected edit, some in a log, some in a half-finished instruction update.

WikiSkill inserts a separate memory layer between logs and skills. That makes the system do what engineers have long treated as routine: keep not just the final instruction but a record of reasons, exceptions and recurring mistakes.

This matters most for long, tool-heavy tasks, where a failure rarely comes down to one line. For example:

🟣 an agent gets stuck in a loop in a text environment and repeats the same action;

🟣 an agent on a spreadsheet task writes short workaround commands instead of a reliable script;

🟣 an agent working on long documents starts reading everything instead of searching for the passage it needs;

🟣 an agent proposes again an idea that was already tried and rejected.

An ordinary skill can only record the conclusion: do it this way. The wiki also records why the edit was made, which alternatives failed and which symptoms recur from iteration to iteration.

How it was tested

The authors ran WikiSkill across five benchmarks and five models.

The task mix is deliberately varied:

🟠 math — LiveMath;

🟠 web search — SealQA;

🟠 spreadsheets — SpreadsheetBench;

🟠 long-document question answering — OfficeQA;

🟠 an interactive environment — ALFWorld.

The model lineup is spread out too: Qwen-3.5 4B, Qwen-3.5 9B, Qwen-3.6 27B, Gemma-4 31B and Gemini-3.5-Flash.

The comparison covers three skill-evolution systems — EvoSkill, Trace2Skill and SkillOpt — plus a baseline with no skills at all.

The authors ran the full evolution cycle three times for each method to smooth out the noise of small validation sets. When validation is small, a single lucky or unlucky example can swing the decision to accept or reject an edit.

The results

WikiSkill consistently beats both the no-skills baseline and the other skill-evolution methods.

WikiSkill outperforms the no-skills baseline and other skill-evolution methods; the gap widens on stronger models.

Average accuracy across the five tasks moved like this:

🟣 Qwen-3.5-4B: from 26.2 to 38.5

🟣 Qwen-3.5-9B: from 29.9 to 47.4

🟣 Qwen-3.6-27B: from 39.4 to 63.3

🟣 Gemma-4-31B: from 41.3 to 54.9

🟣 Gemini-3.5-Flash: from 49.5 to 68.1

The numbers in short:

🟠 the stronger the model, the more it gains from skill evolution;

🟠 within the Qwen family the gain was 12.3, 17.5 and 23.9 percentage points for the 4B, 9B and 27B models respectively;

🟠 model scale and skill evolution add up rather than substitute for each other.

The common assumption is that skills are a crutch for weak models. The data says otherwise: bigger models extract more value from good procedural instructions.

A few more observations:

🟠 a small model with skills can beat a large one without them: Qwen-3.5-9B with WikiSkill scored 47.4, while Qwen-3.6-27B with no skills scored 39.4;

🟠 spreadsheet tasks show the sharpest jump: Qwen-3.6-27B went from 40.8 to 81.7;

🟠 on math the improvement holds across every model: Gemini-3.5-Flash, for instance, rose from 33.0 to 72.6;

🟠 not every task responds equally: gains on OfficeQA are smaller, and the small Qwen-3.5-4B even slips a little.

That last point matters too. Long documents demand not just a good skill but the discipline to execute it. A small model can be handed a sensible instruction and still fail to follow it through.

Skills transfer between models

One of the most interesting results: skills transfer between models, sometimes working better than they do for the model that produced them.

A skill developed by Qwen-3.6-27B lifted Qwen-3.5-9B to 50.5 on SpreadsheetBench. For comparison: 24.3 with no skills, 33.6 with skills of its own.

It runs the other way too: skills found by a small model helped a bigger one. Which points at something important: discovering a useful procedure and executing it well are different abilities.

But transfer is not always safe. The authors also show negative transfer. On spreadsheet tasks, skills from Qwen-3.5-4B dropped Gemini-3.5-Flash from 50.5 to 18.1.

Why that happens:

🟣 a small model writes down workarounds that are too low-level;

🟣 that costs the stronger model the freedom to pick a more general and more reliable route;

🟣 fragmented instructions lead to unnecessary tool calls;

🟣 the model runs out of its step budget before finishing the task.

In short:

🟠 a skill is a portable artifact, but not a universal one;

🟠 cross-model transfer has to be validated separately for each task type;

🟠 a skill's quality depends not only on who executes it but on who found it.

Why the wiki layer is the part that matters

The authors tested separately what the persistent knowledge base actually contributes, as opposed to simply adding one more analysis step.

When the module that proposes new skills has no access to the wiki, the average score on Gemini-3.5-Flash is 48.7. With access, it climbs to 63.7. A difference of 15 percentage points.

The gain comes down to one thing: the system remembers the context of past attempts.

An ALFWorld example: a rejected edit does not disappear — it stays in the wiki and helps shape the next, successful version of the skill.

The ALFWorld case makes it concrete. The system first spots an action loop and proposes a fix; the fix fails validation and is rolled back. But the record of that failed edit stays in the wiki. On the next iteration the module sees that this route has already been tried and did not help. So it writes a different, more specific skill — this time with a rule of behavior that holds.

The log becomes working knowledge for the next step.

What changes in practice

WikiSkill pushes the conversation about AI agents somewhere useful: how to organize the accumulation of procedural knowledge outside the model weights.

A few practical conclusions follow:

🟠 not everything needs fine-tuning — some improvements are easier to keep as external skills and a knowledge base;

🟠 memory of failures has to outlive a single iteration;

🟠 strong models gain the most from good scaffolding;

🟠 cross-model skill transfer is real, but it has to be checked per task type;

🟠 the split “raw logs → knowledge → skill” works better than dumping everything into one prompt or one instruction file.

There are limits. In the paper, skills are simply dropped whole into the system prompt, so retrieving the right skill from a large library remains an open question. Validation is strict: only edits that improve the score immediately are accepted. And the wiki itself can grow over time until it needs pruning.

The takeaway

WikiSkill makes a simple point: if you want an AI agent to improve across iterations, a history of runs is not enough. You need a persistent knowledge base that holds failure patterns, procedures that work, and a memory of attempts that did not.

Three ideas come out of the results:

🟣 skill evolution works better when knowledge is kept separate from the skills themselves;

🟣 stronger models get even more out of such skills;

🟣 skills can move between models, but the transfer depends on how general the procedures turned out to be.

In short, the future of AI agents looks like more than a race for bigger models. It also looks like engineering discipline around them: logs, memory, careful edits, tested hypotheses, and scaffolding that knows how not to forget.