Distilling 1,000 GitHub repos into agent skills more than doubles MLE-bench scores

When an agent lacks experience

AI agents can already write code, run experiments and even attempt to reproduce papers. But on long research tasks they keep hitting the same problem: they know the general ideas but are bad at getting them into working shape.

That is the difference between "I've heard of this method" and "I know which package to install, what data format it needs, where the data pipeline will break and how to check that before you burn half your GPU budget."

The authors of Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills call this missing layer operational knowledge. The idea: the agent has a model, and it has a scaffold that manages steps, memory and checks. What is missing between them is one more layer — ready-made, verified skills, assembled from repositories and papers so they can be used in the middle of a task.

The numbers are hard to ignore: add such a skill library to the same GPT-5.5-based agent and performance rises by 134.3% on MLE-bench, 34.4% on PaperBench, 9.2% on FrontierCS and 14.0% on PassNet.

Talk about agents usually comes down to two knobs: pick a stronger model, or write a smarter scaffold. This paper points at a third: hand the agent pre-assembled, pre-verified working experience.

What the authors mean by a skill

In this work a skill is not an abstract "capacity to reason". It is a concrete artifact the agent can open and use.

A skill has three parts:

🟠 A short instruction: what the skill is, when it is needed, how to use it, and the usual mistakes.

🟣 Reference material: details on the API, settings, limits, parameters, examples.

🟠 Scripts: ready-to-run wrappers, so the agent does not rewrite routine pieces from scratch.

The key point: the agent should not have to work out from scratch every time how a package is built, what its interface looks like and how to run the standard pipeline. That knowledge can be compressed, structured and verified in advance.

The idea of the paper: skills add the missing layer between the model and the scaffold and improve the agent's results across several benchmarks.

The authors also draw a distinction between two parts of knowledge:

🟠 Knowledge as capability: which tools, methods, models and APIs are available at all.

🟣 Knowledge as policy: when exactly to use them, why, in what order and with what checks.

Without the first, the agent has no tool. Without the second, it has the tool but no sense of whether this is the moment to reach for it.

What DisCo is

The system is called DisCo. It is one agent in two roles.

In the first role it builds skills. In the second it uses them to solve a research task.

Building a skill takes four steps:

🟠 Identify the capabilities: work out what a repository can actually do, or what a specific task requires.

🟣 Gather the evidence: pull from the source, the documentation, the tests and the demos everything that genuinely confirms those capabilities.

🟠 Package it as a skill: write the instruction, the reference material and the scripts.

🟣 Verify it: run the tests, safe examples, interface checks and smoke tests — and when something does not line up, fix it locally and check again.

The authors are not just summarizing a repository. They are trying to turn human documentation into working, checkable knowledge for an agent.

The DisCo pipeline: the agent first extracts and verifies operational knowledge, then loads only the relevant branch of skills while solving the task.

There are two build modes:

🟠 Task-independent mode: run popular repositories and papers through the pipeline ahead of time and assemble a library of reusable skills.

🟣 Task-oriented mode: for a specific task, work out which knowledge is missing, find the sources and build the needed skills quickly.

The first gives you a long-lived library. The second lets you top it up with what this task needs right now.

A library from 1,000 repositories

On top of this pipeline the authors assembled the AREX-Skill Library — 5,353 skills extracted from 1,000 popular repositories in machine learning.

Those skills are sorted into:

🟠 20 domains

🟣 178 capability families

So the agent does not get everything dumped into context at once. It goes through a router: domain first, then family, then the skill graph for the specific repository.

How the skill library is built: more than 5,000 skills from 1,000 repositories, plus a router that helps the agent load only the fragment it needs.

This solves two problems at once.

First, the context does not overflow. The agent does not have to read a whole repository or the whole skill archive.

Second, the knowledge becomes reusable. Once the agent has neatly packaged working experience for, say, `vLLM`, `sentence-transformers` or `AlphaFold`, that experience carries into dozens of later tasks.

The short version:

🟠 Not the whole repository in the prompt, but a short entry skill and the branches you actually need.

🟣 Not a one-off hint, but a durable artifact you can test and update.

🟠 Not memory of the last run, but a library of operational knowledge.

What the benchmarks show

The most striking result is MLE-bench, a set of 75 tasks in the spirit of machine learning competitions.

The authors compared the same Codex agent on GPT-5.5 in two modes: without skills and with them. The budget per task was held constant. The only thing that changed was access to pre-assembled operational knowledge.

The result:

🟠 31.11% → 72.89% on the Any Medal metric across the full set.

🟣 +41.78 percentage points in absolute terms.

🟠 +134.3% relative to the version without skills.

What is especially interesting is that the gain is larger on the hard tasks:

🟣 13.33% → 62.22% on the hardest subset.

The harder the task, the more expensive trial and error becomes for the agent. If it already knows which libraries to look at, which settings usually work and where the standard traps are, it reaches the workable region of solutions faster.

On PaperBench, where the job is to reproduce papers, the average score rose from 29.45% to 39.59%. The improvement there is less dramatic but consistent: skills helped on 18 of 20 tasks.

On FrontierCS the gain was 9.22%. In raw numbers that is more modest, but a different effect matters there: skills most often rescued the tasks where the agent without them got stuck at a weak result. The authors show separately that the improvement is not explained by simply spending more tokens or more steps. The correlation is close to zero. The agent was not just "thinking longer" — it was thinking with more direction.

On PassNet, where the agent has to generate optimizing transformations for a graph compiler, skills lifted the headline score from 1.343 to 1.5313 and cut the number of failed examples from 14 to 5. Here a skill helps less with finding a cleverer optimization than with not breaking correctness and not getting lost in the checks.

The short version of the results:

🟠 MLE-bench: +134,3%

🟣 PaperBench: +34,4%

🟠 FrontierCS: +9,2%

🟣 PassNet: +14,0%

Why it works

Most failures of research agents come down to a shortage of applied working knowledge.

An agent can understand that it needs a fine-tuned transformer, a different validation setup or a particular inference mode. But then come the small details, and that is where everything falls apart:

🟠 the wrong input data format;

🟣 an incompatible package version;

🟠 an incorrect API call;

🟣 budget wasted on a run that was never going to work;

🟠 missing checks that would have caught the error earlier.

People rarely derive all of this from scratch in real work either. We lean on accumulated experience: we read other people's repositories, we remember the standard pipelines, we copy working templates, we keep a mental list of the usual breakages. The authors are doing roughly the same thing, only trying to turn that experience into a form a machine can use.

Where the approach runs out

Strong as the results are, the limits are visible too.

First, building skills costs money and time. For the repository library the authors spend around $40 per repository on average. Not astronomical, but at scale it adds up.

Second, quality depends on routing. Load the wrong skill, or too general a skill, and it can pull the agent away from a better strategy. On PaperBench there were two tasks where the version with skills did worse than the baseline.

Third, knowledge goes stale. Repositories change, APIs move, best practices shift. Which means the skill library has to be maintained and periodically rebuilt.

But all of that reads as engineering constraints, not as a refutation of the idea.

Takeaway

Operational knowledge is what the conversation about research agents has been missing for a long time. The model supplies general capability. The scaffold manages the steps. Skills add working experience: what to run, in what order, with what checks and which traps to watch for.

A few things follow from that:

🟠 Agents can be improved by more than a new model and a new scaffold. A separate layer of reusable knowledge buys a large gain on its own.

🟣 The harder and longer the task, the more pre-built skills pay off. On hard tasks the price of trial and error is simply too high.

🟠 Repositories and papers can be turned into machine-readable working experience. Not a summary, but instructions, checks and executable wrappers.

🟣 The next stage for research agents is skill libraries, not just bigger models. If you want an agent to do useful work reliably, it needs more than intelligence: it needs an accumulated way of working.

This is, in effect, the move from "an agent that learns to live from scratch every time" to "an agent that arrives at a task already carrying a set of proven working habits". For AI research that may matter more than another turn of the crank on multi-agent scaffolding.