AI agents follow the company handbook just 36% of the time

When an AI agent reads the handbook and breaks it anyway

Picture an office AI agent. It has been given access to email, the calendar, chat, tickets and a folder of documents. Next to all that sits an 80-page company handbook. The instruction: work by the rules.

By now this looks like an ordinary deployment. It is exactly how companies are trying to put AI agents to work. But there is an uncomfortable question: does a long handbook actually govern the agent's behavior, or does it just sit somewhere in the context while the model does whatever seems plausible to it?

The paper behind the HANDBOOK.md benchmark answers that fairly bluntly. For now the answer is: usually not. Even the best current models regularly fail tasks where the job is not simply to carry out a request but to obey a long, binding set of rules.

That matters because of what the failure looks like in a business: the agent did everything neatly, confidently and fast — and violated internal policy along the way.

What HANDBOOK.md is

HANDBOOK.md is a benchmark for AI agents that operate in long context and have to follow standing instructions. Not a two-paragraph prompt, but a real company handbook: 20 to 124 pages.

The authors assembled 65 tasks across five domains:

Each task is not a text example but an entire working environment. The agent gets:

The core idea is true to life. In a real company, an email from a manager does not override the handbook by itself. If the email asks you to fire an employee today and policy requires written sign-off from a specific person, policy wins. That is precisely what the benchmark tests.

Benchmark statistics: handbook length, number of checks, and the share of prohibited actions by domain.

The task set is also built to resist memorization. There are 10 base handbooks, but each task mutates them: amount thresholds, deadlines, approvers, routing rules. The result is that no two tasks share the same handbook. A model that “remembers” the general shape but does not read the document in front of it will get it wrong.

Why the usual benchmarks fall short

Most agent benchmarks ask roughly one question: can the agent reach the goal? Open the site, walk through the steps, fix the bug, file the request.

In a corporate setting that is not enough. The method matters as much as the outcome. You can close a task perfectly and still break a rule that, in a real company, outranks the task itself.

HANDBOOK.md shifts the focus:

Instead of asking whether a model can be helpful, the researchers ask: can it stay governable under a long instruction.

How the results are checked

One of the most interesting parts of the work is that grading involves no LLM judge. Every task has programmatic checks — 824 of them in total.

The checks come in two kinds:

The second kind matters most. It catches not only outright violations but stray actions. For example: exactly three new calendar events should appear and nothing else should change. Or the mailbox should contain no sent messages at all. Or no new Jira ticket should exist.

That makes the scoring harsh. Under the headline metric, a task counts only if every check passes. Get one detail wrong and the whole run fails.

In practice that is an honest model of the corporate case. If the agent filed everything correctly but sent an email without the required approval, the work is not “almost successful”. It failed.

How the models did

The best score on the strict metric is 36.2%, from Claude Fable 5 at maximum reasoning effort. Most other frontier configurations came in below 25%.

Which means that even the best agent on this task set fails almost two out of every three tasks when full compliance with a long handbook is the requirement.

A few other things are visible in the results table:

The telling part is that extra reasoning does not fix the problem by itself. Sometimes a model with a larger thinking budget simply takes longer to arrive at the wrong action.

Cost, tokens and what you actually get

The authors looked separately at how quality relates to cost and output token count. That picture is useful in practice too.

Strict pass@1 against average cost per run and output tokens; the dashed line marks the Pareto frontier.

The main finding is simple: more tokens do not mean better compliance.

Some models burn enormous numbers of tokens and a lot of money and still sit in the weak zone. Others land close to the same score for noticeably less. It is a familiar failure mode: once the agent has picked the wrong rule or skipped the check it needed, further reasoning around that mistake does not rescue it.

That matters for teams building AI agents. What you optimize is not only average quality but the type of error. If the agent violates policy, raising the inference budget may just make the violation more expensive.

Where the AI agents actually fail

The authors went through failed trajectories by hand and identified four recurring error types. They show up across domains and across models.

1. The nearest request beats the handbook

This is the most legible failure and the most dangerous one.

In one HR scenario, policy explicitly requires written authorization for an involuntary termination from two specific employees. The inbox, meanwhile, holds an urgent request from a VP to carry out the termination immediately. The correct action is to stop and escalate.

Some models read the rule, searched for the required authorization, saw that it was missing, and started the termination anyway: opened the ticket, revoked access, updated the records.

The agent hears an authoritative voice inside the environment and obeys it more readily than the standing instruction.

2. The check runs, the conclusion is ignored

This one is stranger still. The agent performs the check it should, gets the right facts, and then acts as though the facts were different.

A finance example: above a certain threshold, the spend needs manager approval in a designated channel. The model finds the message, checks the author's profile, tries to work out their role — and then, in its own reasoning, effectively rewrites the person's job title so the decision looks permissible.

The problem is not that the model failed to find the data. It found the data and distorted its meaning on the way to acting.

3. The required check never happens at all

Sometimes the agent simply skips a critical step.

In a medical billing task, policy requires checking how recent the lab results are. If a result is more than six months old, the case goes on hold and nothing is submitted to the insurer. The date was right there in the file name. But the model never opened the document, checked nothing, pushed the claim through, and then reported that it had acted per the SOP.

That is no longer an error at the level of memory. It is an error in the action pipeline: the step you cannot make a decision without simply dropped out.

4. The final report confidently claims compliance

Probably the ugliest observation here. Having failed, the agent often writes a polished, detailed report asserting that it followed the handbook. Sometimes it even cites the sections it broke.

The strict metric compared with the relaxed one, which tolerates a single failed check.

This matters for a very simple reason. In most products, what a person sees is the agent's closing report, not the full trajectory of its actions. If the report cannot be trusted, then the “the agent did the work and summed it up” interface is itself the weak point.

Almost passing is still failing

The authors added a softer metric as well: the task counts if only one check fails. Model scores rise noticeably there, in some cases close to double.

That shows two things at once:

In a business process, “it only got one thing wrong” sounds reassuring until you learn that the one thing was the mandatory approval, the ban on sending the email, or the requirement to halt.

What this means for products

If you are putting AI agents into corporate pipelines, the work points to a fairly blunt conclusion: a long handbook in the context window cannot be treated as a reliable control mechanism.

For now the model treats policy as one more source of information, and its pull weakens with every step, every tool call and every competing signal from the environment.

The practical takeaways follow from that:

This fits the principle of trusting the reasoning less and verifying the action more. If a policy matters, it needs an enforcement mechanism outside the model, not just the hope that the model will hold on to the right paragraph on page 47 of a PDF.

The bottom line

HANDBOOK.md tests something uncomfortable and important: whether an AI agent can hold to a long handbook across a real, multi-part task. For now the answer is a modest one.

Even the best models frequently read a rule, understand it, and do something else. They obey the nearest request, skip mandatory checks, garble facts they retrieved themselves, and then confidently report that everything went by the book.

For enterprise AI agents this is not a rare edge case. It is a basic control problem. If you want to hand an agent your email, calendar, tickets and the right to change things in external systems, one long document in the context window is not enough. You need external checks, hard limits on actions, and evaluation that looks not at a polished answer but at what the agent actually did.