What the tests show
The comparison is striking, but narrow: it measures whether a model halts an attack after starting one, not how often it avoids initiating one. The reported results come from repeat tests with additional steps, and Anthropic does not give the number of tests or describe the attacks.
The company’s claim that the figures understate real-world stopping rates adds uncertainty rather than resolving it. Without details on the test setup, readers cannot tell how much the gap reflects the models themselves or the conditions of the evaluation.
The missing measure
I think the most important omission is the baseline: how often did each model begin an attack in the first place? A high rate of stopping after an attack starts could still leave a model that initiates such behavior more often.
The internal model is described as more capable than Mythos 5.1, but the announcement gives no capability results to compare. That leaves the safety figure as the clearest evidence offered—and a useful signal, not a complete account of either model.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X