Compiling a paper into a repo-level spec cuts AI's algorithmic shortcuts

When AI writes code from a paper, it cuts corners too often

Picture the task: you hand an AI a machine learning paper and ask it to build an entire repository from it. Not one file — a real project: the model, data loading, training, evaluation, run scripts. On paper it sounds straightforward. In practice the AI almost always starts simplifying.

Somewhere an important detail of the algorithm goes missing. Somewhere one file expects a tensor in one layout and another produces a different one. The result looks plausible, but the method is no longer the one in the paper.

This is exactly the problem PaperCompiler sets out to solve. The authors propose treating paper-to-code as compiling a specification. First the system pulls the requirements out of the paper. Then it binds them to specific files and interfaces. Only after that does it write code.

It sounds like bureaucracy for an AI agent. But that scaffolding layer appears to be exactly what you need if you want a repository that preserves the logic of the method.

Where ordinary pipelines fail

The authors start from an observation familiar to anyone who has pointed a coding agent at a long task. Between reading the paper and generating code there is usually an intermediate step: a plan, a short summary, a task list, the model's own reasoning. The problem is that the next stage of the pipeline can ignore that summary, restate it in its own words, or simplify it.

What gets lost in that setup is the crucial part: which requirements from the paper must not be weakened.

Three approaches to paper-to-code: general AI agents, specialized pipelines, and the PaperCompiler scheme with a repository-level specification.

PaperCompiler is built around a simple idea:

🟠 The paper should turn into an explicit specification

🟠 Every important requirement has to be bound to the place in the repository where it is implemented

🟠 The links between files must be defined before code generation, not left for the model to guess

This matters because most of the failures in these tasks are not syntactic. They are semantic. The code can run and still not match the paper.

What PaperCompiler is

PaperCompiler is a pipeline that generates a repository from a paper, with the specification at the center of it. The authors split the work into three large steps.

The first step is grounding the paper. The system turns the paper into a structured description of what actually has to be implemented: the core model, data processing, training, evaluation, external dependencies. And instead of writing a plain summary, it tags the status of every fact:

🟣 Confirmed by the paper

🟣 Inferred from indirect evidence

🟣 Delegated to an external source, such as another repository or protocol

🟣 Left explicitly unresolved

In ordinary AI pipelines all of this gets blended together. Here the system at least knows what it took from the paper and what it filled in itself.

The second step is compiling the specification. Here the facts stop being a description and become requirements:

🟠 What must be preserved without simplification

🟠 Which substitutions are forbidden

🟠 Which file is responsible for what

🟠 Which artifacts one module produces and another consumes

🟠 Which constraints have to hold across files

The third step is constrained repository generation. Code is written file by file, in dependency order. Each file receives not the whole paper but a local specification: what it must do, which interfaces it has to expose, what it must accept from other files, and what it may not simplify.

The full PaperCompiler pipeline: parsing the paper, assembling the specification, assigning responsibility to files, and generating the repository in dependency order.

In short, instead of one big prompt saying "build a repository from this paper," the authors make the coding agent work against contracts.

A telling example: when the algorithm quietly gets simpler

The paper includes a case study on Universal Neural Functionals. A key part of that method is constructing basis elements over all admissible partitions. Keep a single candidate and the method changes.

In this example the standard PaperCoder-based pipeline makes exactly that cut: it effectively keeps one partition and throws away most of the basis. From the outside the code still looks like the code you wanted. Inside, it is a different implementation.

PaperCompiler holds on to that requirement and generates code that really does enumerate the admissible partitions and assemble the corresponding basis blocks.

The Universal Neural Functionals case: the ordinary pipeline collapses an important rule of the algorithm, while PaperCompiler preserves the enumeration of admissible partitions.

This is a clear illustration of what the paper is about. The failure is not that the model forgot a line. The failure is that semantically, the algorithm became simpler than the paper requires.

For research code that is fatal. Especially if someone later uses the repository as the basis for experiments, or as a "reproduction" of the method.

How they evaluated it

The authors tested the system on 90 papers from Paper2CodeBench and an additional P2C-Ex set. These are papers from ICLR, ICML and NeurIPS 2024. The comparison covered not only general multi-agent systems like ChatDEV and MetaGPT, but also specialized paper-to-code systems: PaperCoder, AutoP2C and AutoReproduce.

Scoring ran in three modes:

🟣 Reference-free — the judge sees only the paper

🟣 P2C-Ex — a more detailed reference-free evaluation

🟣 Reference-based — the judge compares the generated code against the authors' repository

The last mode is the interesting one. It shows best whether the system really hit the implementation details or merely assembled a convincing facade.

What the numbers came out to

Against the closest comparable baseline, PaperCoder, the new approach raises the average score:

🟠 from 4.562 to 4.777 in the reference-free evaluation

🟠 from 4.535 to 4.728 on P2C-Ex

🟠 from 3.647 to 4.152 in the evaluation against the authors' code

The number that matters here is the last one. That is a 13.8% relative improvement in precisely the mode where semantic errors are hardest to hide.

The numbers in brief

🟣 The biggest gain comes in the comparison with the authors' implementation

🟣 13.8% relative improvement in the strictest mode

🟣 PaperCompiler preserves the details of the method, not just the appearance of a repository

Per-paper win rates on a ten-paper subset: PaperCompiler beats AutoP2C, AutoReproduce and PaperCoder more often, especially in the comparison against the authors' code.

In plain terms: PaperCompiler is better not only at producing a complete set of files. It is better at not distorting the method itself.

The authors also break down the error types. With PaperCompiler, the share of severe findings from the judge drops noticeably:

🟣 Severe errors: from 13.2% to 6.1%

🟣 Algorithmic simplifications: from 28.0% to 24.6%

🟣 Missing key components: from 12.3% to 6.8%

🟣 Evaluation protocol mismatches: from 13.4% to 8.4%

So the progress is not only in the average score but in the character of the failures. Repositories less often end up looking roughly right with something critical broken inside.

The results in brief

🟠 The biggest gain is where the comparison is against the authors' implementation

🟠 Severe errors cut roughly in half

🟠 Fewer key parts of the method left out

🟠 Fewer mismatches between files and pipeline stages

Why it works

The interesting part here is not the bump in scores but the mechanism.

PaperCompiler does three things coding agents usually lack:

🟣 It records explicitly which requirements may not be degraded

🟣 It assigns an owner to every requirement — the file responsible for implementing it

🟣 It describes how artifacts pass between files before any code is generated

This sounds a lot like ordinary engineering. That is the point. Many paper-to-code pipelines still count on the LLM holding everything in its head: the details of the method, the project structure, the dependencies, the interface constraints. On short tasks that sometimes works. On repositories it does not.

The authors also run ablations, switching off parts of the system one at a time. The sharpest drop in quality comes when they remove requirement alignment and file-level contracts. Those two layers matter most for matching the authors' code.

That matters. The gain here does not come from simply handing the system more tokens. AutoP2C, for instance, spends a comparable token budget and still loses by a clear margin. So what decides is not the budget alone but the way information is organized inside the pipeline.

Where the problems remain

PaperCompiler does not solve the problem outright.

First, the system relies mostly on text-based parsing of the paper. If an important detail lives in a complicated figure, a visual example or an architecture diagram, the pipeline may fail to recover it.

Second, the specification helps during generation but gives no formal guarantee of correctness. A repository can be well put together structurally and still fall over on external dependencies, rare cases, or simply running at all.

Third, PaperCompiler picked up slightly more findings about APIs and data schemas. The system holds the method better but can still get the finer interface plumbing wrong.

Finally, the approach costs more compute. On average it spends 1.71 million tokens per repository against 0.98 million for PaperCoder. In absolute dollars, by the authors' estimate, that is still a moderate price for a visible quality gain.

Where the limits show most

🟣 Text-based parsing can miss meaning carried by figures and diagrams

🟣 A specification does not guarantee the code runs correctly

🟣 Errors in APIs and data schemas are still there

🟣 The approach demands more compute and more tokens

Why this matters

On a long task, quality increasingly depends less on the model than on the scaffolding around it.

When the task is small, you can hope one good prompt will carry it. When the task is a repository built from a research paper, writing code is no longer enough — you also have to hold on to commitments:

🟠 what the paper actually says;

🟠 what we filled in ourselves;

🟠 what must not be simplified;

🟠 where in the project it belongs;

🟠 who passes what between files.

This goes beyond paper-to-code. The same logic matters for AI agents in analytics, robotics, research automation and large software tasks. The longer the pipeline, the more expensive it is to lose meaning between the steps.

The takeaway

Turning papers into code more accurately takes more than giving the model more context and more tokens. You have to build the repository-level specification first, and write the files after that.

That approach visibly reduces algorithmic simplifications, cuts the number of severe errors, and preserves more of the method's details in the comparison against the authors' code. In long engineering workflows, progress comes not only from the model but from how the pipeline stores, passes and defends semantic requirements.

If you want AI to do more than generate plausible code — to actually reproduce the method — going without specifications like these will only get harder.