Auto-generated AGENTS.md files make coding agents worse and costlier

The idea looks obvious: give a coding agent a dedicated file with the repository's rules — how to build the project, how to run the tests, what the conventions are for style and structure — and it will work more confidently and make fewer mistakes. Such files are usually called context files: AGENTS.md, CLAUDE.md and the like. There are already tens of thousands of them in open source, and many agent frameworks ship an “init” command that generates one for you.

The problem is that almost nobody has tested the practice rigorously. “Add an AGENTS.md and your agent gets smarter” sounds convincing, but it is unclear whether that context actually helps on hard repository-level work, where the agent has to find its way around someone else's code, fix a bug or add a feature without breaking the tests. The authors of Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? set out to answer that experimentally — and the results are unexpected.

How the evaluation works and why it is closer to reality

The authors test context files in two worlds. The first is the familiar popular repositories from SWE-bench Lite, which usually have no such file, so one has to be generated with an LLM following the recommendations of the specific agent. The second is more interesting: a new benchmark, AGENTbench, built from less “starred” but living Python repositories where the developers added an AGENTS.md/CLAUDE.md themselves and actually use it.

AGENTbench is put together in a down-to-earth way: real GitHub issues and the pull requests that closed them are used to reconstruct the task and a test-based check. The result is 138 tasks across 12 repositories — some bug fixes, some new functionality. One detail matters here: since many small projects ship PRs without tests, the authors generate regression tests that fail on the “before” version and pass on the “after” one, then manually prune the cases that turn out too strict.

Does AGENTS.md help agents solve tasks better

The headline result is close to a paradox for the industry's own advice: automatically generated context files hurt more often than they help. Averaged across settings, they produce a small drop in the share of solved tasks (a few percent), and the effect holds across different agents and models.

Human-written files — the ones developers actually committed to the repository — look better: on AGENTbench they give a small gain over the no-context mode (about +4% on average). But a small gain is exactly what it is, not a breakthrough. And, importantly, even human files almost always raise the price of an attempt.

The price of hints: the agent tries harder, and does not always win

Once a context file is in the repository, agents behave differently. They explore the codebase more, run tests more often, use tools more heavily, and follow repo-specific instructions more often — switching to uv, say, when the file tells them to. At the level of behavior it all looks right: the agent is more disciplined, more careful, more “engineering-minded”.

But that may be precisely what causes the drop: extra requirements expand the space of mandatory actions. The agent spends more steps and more reasoning tokens, and ends up either running out of its iteration budget or wandering off into an investigation that was not the one it needed.

In numbers it is blunt: inference cost rises by more than 20% almost everywhere, and the step count goes up consistently. A context file, in other words, is close to a guarantee that the attempt costs more, even when success does not improve.

Why a map of the project does not save it

Many context files include a tour of the project structure: a list of directories, notes along the lines of “code lives here, tests live there”. It seems like this ought to speed things up. The authors check it with a simple but telling proxy signal: how many steps pass before the agent first touches a file that the reference PR actually changed.

And here is the surprise: without context files that moment often comes sooner. A guided tour of the repository does not get the agent to the right places faster — sometimes it does the opposite.

The one setting where autogeneration does pay off

There is an important caveat that makes the paper unusually honest. The authors test a hypothesis: maybe LLM-generated context files look harmful only because popular repositories are already heavily documented, and the file simply restates what is already there?

So they strip the documentation out of the codebase (md files, docs directories and so on) and rerun the evaluation. In that desert, autogenerated context files start winning, and sometimes beat the human-written ones. That neatly explains why developers often do feel a benefit in practice: in a poorly documented project any coherent quick guide genuinely helps.

What this means if you are the one writing AGENTS.md

The paper does not tell us to stop using context files. It resets expectations and points to a style: the more requirements the file carries, the likelier it is that the agent gets slower and more expensive — with no guarantee it gets more accurate. The best human-written AGENTS.md, judging by these findings, is the minimal one: only what the agent genuinely cannot start the project without — commands, the test runner, key environment variables — rather than an attempt to retell the architecture and lay down rules for every occasion.

One-click autogeneration is something the authors explicitly do not recommend: on average it lowers the solve rate and reliably raises cost. If you are going to generate a file at all, treat it as a stopgap for documentation that does not exist — and then have a human edit it, cutting everything that is not needed.