MLE-Smith auto-generates 606 ML tasks that rank agents like human benchmarks

When the subject is measuring what AI can do in machine learning engineering, the default answer is a static benchmark: a competition assembled once by its organizers, a single dataset, a fixed metric. That is convenient, but it scales badly. Every task has to be verified at length, forced into a common format and updated by hand. What you end up with is a narrow world of tasks, while the real supply of data is far more varied. MLE-Smith proposes to break the pattern: turn raw datasets into live competition tasks automatically, keeping their structure, their meaning and their practical value.

Why this matters

Static suites hold progress back for three reasons. First, they go stale fast: the industry moves, the benchmarks do not. Second, they cover modalities and domains narrowly, because preparing tasks by hand is expensive. And third, such tasks tend to lose realism — the rough edges an engineer actually runs into have been scrubbed out. Generate tasks automatically and you can keep opening up new regions of the data space while holding the quality bar high.

How the pipeline works

At the core of MLE-Smith is a multi-agent system built around a generate–verify–execute scheme. The roles divide up like this:

• Brainstormer studies the raw data and proposes several task formulations, each with objectives, metrics and a feature preparation plan. The governing rule: rely only on what is actually in the data.
• Designer turns an idea into a complete, executable competition package — from the preparation script to the metric, the description and the tests.
• Refactor brings everything to a single standard: a strict directory tree, a sample submission, held-out test answers, an integrity check.

From idea to a working competition

The output is a standard set of artifacts: descriptions, the public portion of the data, private answers for scoring, one shared evaluation protocol. prepare.py assembles train/test deterministically and guarantees no leakage. metric.py validates the prediction format and computes the score. All of it ships with automated tests and strict interface checks, so a task can be run without manual patching.

Verification with no free passes

Verification runs in three layers. First the hard assertions: directory structure, function signatures, a successful run of preparation and metric, valid sample files. Then a soft semantic review: are the statements clear, is the metric appropriate, are there trivial shortcuts to the answer. And finally a trial by fire — a run in the interactive environment, where an agent writes and executes code, gets feedback and improves its solution. Clear all three and the task joins the final collection.

How far it scales and what it costs

The system ran through 300 datasets and produced 807 draft tasks. After every check, 606 tasks across 224 datasets came out verified — nearly three per dataset on average. Preparing one task takes about 7 minutes to generate the artifacts plus up to 10 minutes to run it in the environment (it varies by modality and hardware). Average cost: under a dollar per task. Coverage spans tables, text, images, video, audio and time series; the formulations include classification, regression, ranking, multi-label and even generation. The metrics vary too, from F1 and AUC to NDCG and domain-specific ones.

What the experiments showed

The authors assembled a combined set of 100 tasks: 50 human-written ones from MLE-Dojo and 50 generated automatically. Eight current LLM agents competed in the same interactive environment. The Elo ranking came out all but identical on the real and the generated tasks: correlations close to 1, with the top of the leaderboard matching as well. That is an important signal — the artificially created tasks do not make life easier for the models, and they separate the strong from the weak just as sharply.

The step-by-step dynamics are another interesting detail. The agent works in a loop: request information, write code, run it, fix it. The normalized quality curves rise monotonically from step to step, which shows the tasks are sensitive to engineering progress: better code and a better pipeline mean a better result. That is exactly the behavior you want from realistic MLE scenarios.

What this changes for the community

MLE-Smith demonstrates that an automatic task factory is possible. What counts is not only the volume but the procedural guarantees: structural integrity, semantic correctness and confirmed executability. A factory like this makes it possible to expand domain and modality coverage quickly, refresh tasks without the manual drudgery and, above all, compare agents on a field that stays fresh and fair. When scores on synthetically created tasks are indistinguishable from human benchmarks, the result is a new standard for scalable evaluation and training of agents — including those that rely on a world model and long chains of actions.

A closing note

The authors are not trying to replace live competitions. They are building infrastructure in which new data turns into well-tested challenges without a loss of quality. This is precisely the case where automation does not impoverish reality but widens it: more tasks, more varied, and just as good at measuring what models and teams can do.