DeepCode rebuilds a paper's codebase by managing memory, not context

Over the past year, LLM coding agents really have learned something new: they now handle tests, run commands and get through relatively long sessions. But raise the difficulty — ask an agent to "ship the repository for this paper" — and reality arrives fast. A paper carries a lot of load-bearing detail, scattered across its sections, while the finished project is dozens of files, dependencies, scripts and small hacks. All of it has to be held in mind at once, and an LLM's context is finite.

In DeepCode: Open Agentic Coding, the authors reframe the problem as one of managing an information flow. If the channel is narrow — that is, if the context is small — you cannot keep stuffing the paper text, the conversation history and everything written so far into it forever. At some point the important constraints get lost in the noise, and the agent starts contradicting itself: interfaces don't line up, function and command inputs disagree, files read things that no longer exist, and pipelines simply refuse to run.

Carry less, manage knowledge better

DeepCode is a standalone framework that splits development into three large stages and, inside each one, works to free up context without losing meaning.

The system starts by reducing the paper to a short but formalized summary. The document is split into chunks by subheading, and from then on it is not fed to the LLM whole but loaded locally as needed: when the agent has to recall architecture components, formulas, training details or how results were evaluated. Two roles do the work: one collects the high-level information (what is being reproduced at all, which components to pay attention to), the other pulls out low-level detail. The two are then merged into a blueprint — the final source of truth against which everything else is shaped.

Then repository generation begins, and here the authors make a move that is simple and, from another angle, rather brilliant: instead of maintaining an ever-growing prompt, in which writing each new function or file means dragging along all the previous code, they keep a Code Memory — a snapshot of what exists in the project. It is a card with fields: what the module is for, its interface and its links to other components. When the next file gets written, the agent pulls in only the relevant memory entries instead of a mile of text in the prompt.

CodeRAG. In practice, papers rarely explain how to build a data loader properly, or logging, configs, experiment launching, reproducibility, checkpointing and the rest. DeepCode can pull known patterns from outside repositories as they become relevant and use them in the prompt where that helps.

Then, to make the repository actually work, DeepCode runs static analysis to confirm nothing is missing and everything matches the project. It tries to run the project in a "sandbox", after which it keeps clearing up whatever breaks during execution (dependencies, names, launch arguments, mismatched inputs and outputs, and so on).

How it performs: against agents and against people

All of this was tested on PaperBench Code-Dev: 20 ICML papers where the job is to build a repository and a reproduction script from scratch.

By the authors' account, DeepCode scores 73.5 ± 2.8 on replication score — far above any previous LLM agent (the best prior work was around 43) and above PaperCoder, a pipeline built narrowly for this task (around 51 there).

Against commercial coding agents on a subset of five papers: DeepCode averages 0.854, while Cursor and Claude Code sit around 0.58 and Codex around 0.40.

DeepCode also runs on exactly the same model backend as some of the systems it is compared against, which means the gain comes from the information-management architecture rather than from a different model underneath.

And on a small subset of three papers DeepCode, unexpectedly, even comes out ahead of a group of PhD-level experts on average (72.4 for the experts, 75.9 for DeepCode).

What produced the gain: memory, code retrieval and the final check

The authors' argument is that CodeMem is what does it: the naive sliding-window strategy, where old context is simply evicted, tends to push out definitions before they are done being needed. Structured memory holds interfaces, contracts and dependencies compactly, and the files end up more consistent with one another.

With CodeRAG the picture is more interesting still: a slight gain for strong reasoning models and a noticeable one for lighter models. Adding the third, final verification stage buys another few percent. It is those last few percent that turn almost-working code into a project that can actually live.

Why this matters for research and for engineering

The broader claim DeepCode is making: generating a whole repository succeeds not only on model size but on how attention and memory are managed. Once you have an agent that can compress a specification and hold project state in something other than enormous logs, things start to work and projects stop coming apart at the seams.

For research this matters in particular, because reproducibility is not just a convenience — it is how you quickly check a researcher's next idea and give them feedback. Looking ahead, dynamic planning would be worth having: a blueprint you can update in real time, not only at the start of the iterations.

💾 Code