A new benchmark finds reasoning is AI’s main cloud operations bottleneck

CloudOpsBench is an open benchmark for testing AI agents on common cloud DevOps tasks, from fixing services and deleting resources to restricting access, installing updates and closing projects. Version 1.0 includes 45 AWS tasks prepared by experienced DevOps engineers. Claude Fable 5.1 achieved the best result, completing 60%; GPT-6 Astra solved 53% of the tasks. According to a University of Michigan team led by Eng Chen, about 80% of errors stem from reasoning: agents misjudge the scope of a task, fail to check all the necessary services or stop too soon. Roughly 20% of errors come from incorrect actions, some of which can irreversibly damage infrastructure.

A new benchmark identifies AI’s main problem in cloud operations

No model completes every task, and in most cases the bottleneck is reasoning. | Image: CloudOpsBench

Source: the-decoder.com

Where agents fail

CloudOpsBench tests whether agents can carry out tasks in the cloud, not just know how to solve them. The results show that even the best models fail to complete a significant share of the tasks: Claude Fable 5.1 completed 60%, while GPT-6 Astra completed 53%.

The team led by Eng Chen at the University of Michigan attributes most failures to reasoning. In about 80% of cases, an agent misjudges how broadly it needs to inspect the system, misses important services or stops before the task is actually complete. Around 20% of errors result from incorrect actions; some can cause irreversible damage to infrastructure.

AWS testing and plans for expansion

All tasks run the same way in the real AWS cloud and its emulator. The authors plan to expand the benchmark by adding support for GCP, Azure and OCI. The project is published on GitHub under the Apache 2.0 license, alongside a detailed analysis of errors.

Dataist's take: CloudOpsBench shows that success in cloud operations depends on more than getting individual actions right. An agent needs to check the full context of a task and make sure the work is complete. At the same time, roughly one-fifth of errors stem from incorrect actions, some of which can damage infrastructure beyond recovery.