Only 14.5% of agent context files say anything about security

Today, when we write code with AI, we state the task in natural language and the agent inside the IDE plans the steps itself, writes the changes, runs the tests and tries to carry the job through to a result. This is called agentic coding. But it has a weak spot: the agent has to work out fast how the project is put together, what it is conventional to use in it, how to run the build and the tests, which architectural decisions must not be broken and which constraints matter.

That is why dedicated context files have started showing up in repositories — essentially a README written not for people but for the agent. Usually these are Markdown files like CLAUDE.md, AGENTS.md or copilot-instructions.md. The tool loads them at the start of a session and they work as long-term memory: you do not have to explain the same things over and over.

The authors of "Agent READMEs: An Empirical Study of Context Files for Agentic Coding" decided to look at the phenomenon from above rather than through individual examples: what these files actually look like across thousands of repositories, how often they change and what people usually put in them.

The problem the researchers take on

The problem is simple: there is plenty of practical advice on how to write context files properly, but very few facts of the "what actually works" kind. Tool documentation covers the general points — describe the architecture, the pipeline, the standards — but says nothing about what teams really write and which topics they systematically forget. And if these files genuinely steer an agent's behavior, their quality bears directly on both the efficiency and the safety of the result.

Hence the goal of the study: collect a large sample of such files and describe them as an engineering artifact — by size, readability, structure, edit history and the content of their instructions.

How the data was collected and what was done with it

The authors selected open repositories on GitHub (at least 5 stars) and automatically searched the root for files carrying the official names used by three tools: Claude Code, OpenAI Codex and GitHub Copilot. That yielded 2,303 files from 1,925 repositories — as of today a genuinely large empirical sample for such a new kind of documentation.

Then came the inventory, along several lines. First, the authors measured length (in words), readability, and how people structure the text with Markdown headings. Second, they pulled the commit history and looked at how often the files get edited and in what size chunks. And finally they ran a content analysis: they defined 16 instruction types (Build and Run, Architecture, Testing, Security, Performance and others) and recorded which topics come up most often.

In a separate experiment they checked whether such files can be labeled by topic automatically with an LLM (the paper used GPT-5), since doing it by hand scales badly.

The files turned out long, hard to read and… alive

First impression from the numbers: these "READMEs for the agent" are not short hints at all. At the median, files for Claude Code and GitHub Copilot are noticeably longer than those for Codex.

Readability is more interesting still: the texts are often heavy going. For CLAUDE.md the median FRE score is low enough to land in the very difficult band — roughly the level of academic or legal prose. They make sense to their authors and to insiders, but read badly from outside, which means that over time they risk turning into an unmanageable mess.

Structure, on the other hand, is more or less fine: usually one top-level heading and then a breakdown into H2/H3. Deep nesting is almost never seen.

The main surprise is that this is not static documentation. The files get edited actively: a sizable share changed across several commits, and the changes often arrive in bursts of small additions, while deletions are substantially rarer.

It feels like configuration code: add a new rule, append a run command, tighten the style guidance — and so the file grows bit by bit, with old material rarely cleared out. The authors call this a potential source of "context debt": instructions for the agent pile up faster than the team can keep them clear and free of contradictions.

What gets written down — and what goes suspiciously unsaid

Look at the content and the practical topics lead: how to build and run the project, how the architecture works, which implementation details matter, how to run the tests. That makes sense — the agent needs to start up fast and drop into the groove of a specific repository.

Non-functional requirements, though, are rare. Security and performance appear in about 14.5% of files, UI/UX less often still. So agents are frequently told "how to make it work" and far less frequently "how to make it secure, fast, and harmless to the user experience". And when the constraints are not spelled out, the agent will at best fall back on the LLM's general knowledge and at worst take the path of least resistance.

An interesting detail from the labeling: Debugging surfaced as a category of its own, present in roughly a quarter of the files. People are clearly trying to tell the agent in advance where to look at logs and which flags to switch on, so it does not burn time on blind attempts.

Can this be analyzed automatically

Manual labeling is fine, but the industry needs scale. So the authors tested automatic topic classification with an LLM. On a subset of files the model reached a micro-average F1 of about 0.79. The concrete engineering sections — Architecture, Testing, Build and Run — are recognized best. Vaguer things like Maintainability or Project Management do worse, where the category boundaries blur and examples are few.

That is a useful practical hint: linters and CI checks for agent context files are entirely plausible in the future — flagging blank spots (no Security, say), conflicting rules or stale run commands.

What this means for us

The study leaves the feeling that we are watching a new layer of engineering culture being born. Agent context files are quickly becoming the instrument through which a project formalizes its "world model" for the agent: how the system is built and how to act safely inside it. And for now the industry mostly writes operational hints there (build, run, structure) and has not yet acquired the habit of recording quality constraints — especially around security and performance.

The conclusion is simple: if a team already writes code with agents, these files deserve to be treated as code. They need review, updates when the infrastructure changes, and explicit non-functional requirements — otherwise the agent's autonomy will grow faster than control over the result.

💾 Code