Artificial Analysis widens the open-model gap to 12 points

What changed in the index

Closed tests now account for 60% of the index. Artificial Analysis says the shift should make it harder to tune models specifically for familiar benchmark questions.

The revision also changed the standing of Kimi K3.1 Pro: it ranked fourth when launched, but sits ninth in the new version. Artificial Analysis says the previous methodology may have overstated some models’ results.

The cost of harder-to-game tests

The larger gap may reflect more than benchmark gaming. AI researcher Nathan Lambert argues that many training improvements now come from reinforcement-learning environments built around tasks resembling benchmarks. Consistently performing across several areas, he says, requires mature reinforcement-learning infrastructure that not every lab has.

That leaves the index with a trade-off. Closed test sets make it harder to train directly on known questions, but they also prevent outside researchers from spotting flaws in those questions. Artificial Analysis had to correct ambiguous and invalid items in its AA-Omniscience knowledge benchmark for the revised index.

I think the more important question is whether the new spread separates model capability from the resources required to develop it. A ranking can become harder to game while also rewarding labs with stronger training infrastructure. The index’s larger gap does not, by itself, tell us which effect is doing more of the work.