AI auditors catch violations but swamp reviewers with false alarms

Anthropic Institute researcher Andy Hall tested how well AI models can detect violations in other AI agents’ action logs. At a 5% false-positive rate, GPT-5 detected nearly 98% of cases where agents bent the rules to complete a task, while Gemini 3.8 Flash found barely half. But trying not to miss violations also raises suspicion about harmless actions: in Hall’s example of 100,000 agent sessions a month, an audit would produce nearly 3,000 false alerts against about 2,000 correct detections. He argues that these systems are better suited to estimating how often problems occur than to safety checks, where every instance matters.

AI auditors catch violations but swamp reviewers with false alarms

Hall published his findings on the Free Systems blog. He works at the Anthropic Institute and is on leave from Stanford University; he stressed that the article reflects his personal views. The researcher replicated and extended work by METR, using its MALT dataset. It contains logs of frontier models completing tasks from the HCAST and RE-Bench benchmarks. In October 2025, people reviewed some of those logs for violations. Hall compared their results with assessments by GPT-5 and GPT-5 nano, which METR had already studied, as well as GPT-6 Astra, Opus 4.8 and Gemini 3.8 Flash.

What the auditors found

Hall compared the models at a 5% false-alarm rate: out of 100 logs with no violations, five on average are wrongly flagged as suspicious. Under those conditions, GPT-5 detected nearly 98% of reward-hacking cases, in which an agent games a task to get credit instead of actually solving it. Gemini 3.8 Flash found only a little more than half.

On cases where agents sabotaged their own results, all the models tested detected between 71% and 81%. Sabotage, or sandbagging, means deliberately understating one’s abilities—for example, when a model intentionally performs worse on a task to avoid evaluation or restrictions. Most of these cases, however, were artificially created: METR researchers asked models to sabotage tasks, then removed those instructions from the visible chain-of-thought logs.

Every auditor detected the overt violations. In one case, OpenAI’s o3 was supposed to solve a Rust programming problem. It noticed that it shared a code library with the scoring system and rewrote the scoring function so that any submitted work would count as correct. All the auditing models, like the human reviewers, recognized this as reward hacking.

Reviewers missed more subtle violations. Claude Sonnet 4 used a numerical condition in its code that was explicitly prohibited. None of the auditors flagged the deception: GPT-6 Astra considered it an implementation error.

Share of human-flagged misbehavior detected by AI auditors at a 5 percent false-alarm rate. Based on 47 reward-hacking cases and 1,073 sandbagging cases from METR's MALT dataset. | Image: Andy Hall / Free Systems

Source: the-decoder.com

Newer models do not eliminate false alarms

Newer and more expensive models did not always perform better at auditing. Hall compared results with API request prices, using cost as a rough indicator of a model’s recency and capabilities. GPT-5 substantially outperformed GPT-6 Astra at detecting reward hacking, even though checking 1,000 logs with GPT-5 costs about eight times less.

GPT-5 performed about as well as GPT-6 Astra and Opus 4.8 on sabotage. Hall suggested that Opus 4.8’s weak performance at finding reward hacking could point to a tendency to favor itself: most of the logs came from Claude. But, he said, that cannot be proven. Overall, the comparison shows no clear progress in reducing false alarms. Combining several models also brought no notable improvement.

When auditors are tuned to detect 80% of violations, the best models produce false-positive rates of 2.5% to 3% for both reward hacking and sabotage. Some models do much worse: to catch 80% of reward-hacking cases, Opus 4.8 wrongly flagged 40% of harmless logs as suspicious, while Gemini 3.8 Flash flagged 63%.

In Hall’s illustrative example, at 100,000 agent sessions a month and a 3% false-positive rate, there would be nearly 3,000 false alerts against about 2,000 correct detections. Hall worries that reviewers faced with this stream of signals may eventually stop paying close attention to alerts and start missing real violations.

A second evaluation, using the R-Judge dataset from Shanghai Jiao Tong University, showed a similar trade-off on ordinary agent tasks. At a 5% false-positive rate, Opus 4.8 detected nearly 96% of unsafe actions. If the requirement is to detect 95% of such actions, almost all models keep false alarms below 10%, and Gemini 3.8 Flash’s rate is under 4%. But when the requirement rises to 99%, even the best model, GPT-6 Astra, has a 15% false-positive rate. Gemini 3.8 Flash, which led at the 95% threshold, flagged 81% of safe actions as unsafe in this setting.

More expensive does not automatically mean better: auditor discrimination (AUROC) plotted against API costs per 1,000 logs. A score of 1 means perfect separation, 0.5 means random chance. | Image: Andy Hall / Free Systems

Source: the-decoder.com

The last few percentage points cost the most: share of safe agent actions that auditors wrongly flag as unsafe when required to catch 95 or 99 percent of unsafe actions. Based on 270 safe logs from R-Judge. | Image: Andy Hall / Free Systems

Source: the-decoder.com

Major incidents are hard to review by hand

The incident involving Hugging Face agents shows the scale of the task. A group of OpenAI agents left behind about 1,300 transcripts and 1.2 million file fragments. METR analyzed the data primarily with AI agents and spent about $400,000 on APIs. The models used included GPT-5.6 Sol, one of the systems involved in the incident itself.

METR acknowledged that the model often accepted the agents’ perspective without critical scrutiny. The investigation was also not robust against possible deception by the agent analysts. METR subsequently called for serious incidents to be investigated systematically and, where possible, independently.

Study limitations and user responsibility

Hall’s work is a pilot study. To make the results comparable with METR’s work, he barely changed the original prompt. Fable 5 and Opus 5, however, refused to conduct the evaluation under those instructions. The reward-hacking data covers only 47 cases: each individual case shifts the rate by about two percentage points. Hall plans to study separately whether models assess agents from their own developers fairly.

Hall argues that auditors need models with far fewer false alarms. He also says responsibility for different kinds of agent violations should be defined. One proposal he discusses is giving users more control over their own AI auditors.

Some controls are already available: Codex and Claude Code coding agents let users specify which actions can be carried out without asking for confirmation. Hall believes users who grant such permissions should bear greater responsibility for the consequences. In his assessment, a system in which AI checks only other AI would be strange and could be expensive.

Dataist’s take: The results point to a gap between the ability to catch obvious violations and the suitability of AI auditing for reliable safety checks. With the current volume of false signals, automation could shift the burden to people who must review alerts and decide which ones merit attention. Whether models can impartially assess systems from their own developer remains an open question.