i
Research
Review · 2026-11-17

Why Is It Difficult for an AI Agent to Act in the Real World?

Cover: Why Is It Difficult for an AI Agent to Act in the Real World?

A language model can already run a robot: understand a request, break it into steps and call the right skills. But in the physical world, reasoning correctly is not enough. A decision has to arrive before the situation changes, and success has to be proven by evidence, not by a skill reporting “done”.

The core idea

The field calls this agentic robotics. Instead of training a separate policy for every family of tasks, a general model runs the robot in a closed loop: it watches the world, works out what the person wants, reasons, calls skills, checks the result and changes the plan. The trigger was the release of GPT-6 Astra: within weeks, models with agentic harnesses matched or beat purpose-trained policies on many embodied benchmarks. The paper answers three questions: what such a robot is made of, how to test it fairly, and what still holds it back.

🟠 The decision-maker is a pair, not a model. A foundation model plus a harness — the software layer that assembles its context, gives it tools and keeps the plan, memory and verification. Together they form “System 2”.
🟣 Fast is kept apart from smart. Skills and body control run at their own rates and never wait for the model to finish thinking.
🟠 “Done” does not mean “it worked”. A skill's report is only intermediate evidence; success is confirmed by observation.
🟣 Measure the whole system, on a live clock. The harness, the protocol and a paused simulator move results by tens of percentage points.

What an agentic robot is made of

The reference design splits the robot into three systems. Above them is the person and their intent, below them the body and the physical world, and off to the side sits an event monitor that is not part of the command chain.

What an agentic robot is made of
User
Explicit or implicit intent, preferences, interruptions
S2Foundation model and embodied harness
Runs on demand or on events: interprets intent, decides, orchestrates skills and tools, verifies outcomes
Embodied harness
1Context manager
Observations and events; world, user and body state; progress; history
2Tool interface
S1 skills, S0 interfaces, code execution, search, external APIs
3Planner
Task decomposition, candidate plans, replanning
4Memory
Short- and long-term: retrieve, write, update, delete
5Verifier
Checks each step and the whole task from evidence
⇅ the harness gives context, the model answers with a decision
Foundation model
1Knowledge
Of the world, of physics and robot bodies, of programming
2Understanding
Perception, space, physics, behavior, body kinematics
3Reasoning
Intent, planning, tool use, direct control, recovery, interruption
S1Skills and callable modules
Perception, control policies (VLA, WAM and others), motion planning
S0Body interface and real-time control
Whole-body tracking, inverse kinematics, joint control
Robot body
Sensors (cameras, depth, IMU, force) and actuators (joints, grippers, base)
Physical world
Objects, people and space
Real-time event monitor
Watches at a high rate, without waiting for S2: user commands, body movement, scene changes, execution anomalies
Observations flow from the robot body straight up to S2. S2 can also call S0 directly through the tool interface, but S0 still keeps its own real-time loop.

Anatomy of an agentic robot: S2 — a model plus a five-part harness — makes the decisions; S1 skills and S0 body control run at their own rates, and the event monitor can interrupt S2 or slow the robot at once.

🟠 S2 — the model and the harness. The model brings knowledge, understanding and reasoning. The harness turns its answers into controlled action: the context manager gathers observations and the state of the world, the tool interface exposes skills and code execution, the planner keeps and revises the plan, memory holds what does not fit in the context, and the verifier judges from evidence whether a step succeeded.
🟣 S1 — skills. Perception, action policies such as VLAs, and motion planning. A single “grasp the cup” call may segment the object, choose a grasp, plan a path and execute it.
🟠 S0 — the body in real time. Balance, inverse kinematics, joint control. This loop runs at roughly 50 Hz to 1 kHz and keeps the body stable while S2 is thinking.
🟣 The event monitor. It watches the person, the body and the scene at a high rate, without waiting for S2. It interrupts S2 when an event changes the task, and under predefined safety rules it can stop or slow the motion straight away.

Commands flow down, evidence flows up. The split is functional: one model can play several roles, and S2 can send a target straight to S0 or write code that becomes a skill itself.

How it works: the robot looks for a red cup

The request: “Find the red cup and bring it to the dining table.” S2 breaks it into four subtasks — locate, grasp, carry, place — and runs each through a six-step agentic loop.

The agentic loop
The event monitor can interrupt the loop at any step
S2decides on demand
1Interpret intent
The request plus context, such as where the cup usually goes
2Decide
A subgoal, a skill, a control target or a program
3Invoke
An S1 skill, an S0 interface or an external tool — under its contract
4Execute
The skill runs while S0 keeps the body stable
5Verify
Accepted? Terminated? Does the effect hold? Is the user's intent met?
6Continue or replan
Continue, replan, ask, or report
S0 runs continuously at 50 Hz–1 kHz · S1 while a tool runs · S2 on demand

The agentic loop: six steps from intent to a verified outcome; the event monitor can interrupt it at any step.

First S2 calls perception, and the harness records where the cup is and when it was seen. Then it calls the grasp skill and uses a fresh observation to check that the cup is held securely. Once the robot has put it down, S2 confirms that the cup is standing on the table where it was asked to be. Along the way, the harness keeps a past event, “the cup was grasped”, apart from a current state, “the cup is still held”.

The interesting part starts when something goes wrong.

🟠 The person changes their mind. “Not the red cup, the blue one.” The monitor interrupts S2, but a cancel request alone is not enough: before starting a new action, the harness must confirm that the motion has stopped and the body is in a safe holding state.
🟣 The skill gets it wrong. The grasp reports success, but the wrist camera shows the cup still on the shelf. S2 works out that the handle hid the grasp point, looks at the cup from another side and tries a different grasp instead of repeating the same call.

That is why verification is four separate questions: was the request accepted, has execution ended, is there evidence that the intended physical effect holds, and does that effect satisfy what the person wanted.

How do you test a robot that thinks?

Two months after Astra came out, results had become hard to compare: on RoboDojo alone, reports give it anywhere from 22% to 85% success on the same 42 tasks, depending on the harness and the protocol. So evaluation is split into two levels. The model is tested on its own: does it know, understand and reason — in a single answer, with no tools. And the whole system is tested separately, in a closed loop, with its harness, skills and body.

What to measure in a robot agent
9 dimensions · 53 sub-dimensions
What the model can do
A single answer, no tools
Knowledge — of the world, physics, robot bodies, codeUnderstanding — scene, space, physics, people, its own bodyReasoning — intent, planning, recovery, interruption
What the system adds
The model with its harness, in a closed loop
Tool use — perception, planning, action, codeMemory — objects, places, step order, the userCross-embodiment — the same task on another body
What constrains everything
Safety along the whole action chain
Recognize risk and predict consequencesPrevent harm before actingRespond in time and coordinate a stop
What it costs
Efficiency and cost
Latency and response deadlineObservation freshnessTokens, API fees, robot occupancy time

What to measure in a robot agent: nine dimensions and 53 sub-dimensions; 29 of them depend on the body, and standard tests of language models rarely cover them.

The result is a profile across nine dimensions, not a single score: a success rate cannot tell you how safe, fast or expensive a robot is. To make numbers from different groups comparable, the paper proposes rules:

🟠 Separate the model from the system. Comparing models — fix the harness; comparing harnesses — fix the model.
🟣 Run on a world clock. Runs where the simulator waits while the model thinks and runs where the world keeps moving are reported separately.
🟠 Success is confirmed by an independent checker, not by the model itself and not by a tool's return. Agents tend to declare victory too early, and AI judges tend to accept it.
🟣 Disclose the conditions. What the robot sees, what hints and budgets it gets for reasoning, calls and time, and how many runs there were. Without such a “harness card”, the numbers cannot be compared.

Key numbers

19.8% → 0.7%
safe success on RT-Safe's hardest routes: when the simulator waits for the model — and when it does not
40–8,000×
how much longer GPT-6 Astra takes to make one decision than S1 skills take to update control
22–85%
the spread of Astra's success on the same 42 RoboDojo tasks across reports, driven by harness and protocol
72% → 18%
how often the same six models refused harmful requests: in text — and on a real robot

The first two numbers are about time. When the world kept moving, robots completed the task almost as often, but collided 12.3 times more on the way: on the hardest routes Astra spent 7.7 seconds per decision, and by then the situation had changed. Hence the paper's central principle: not “think longer” but get smarter within the deadline — leave fast reactions to the skills and the body, and give the model only what cannot be solved faster. The third number explains why other groups' percentages cannot be set side by side without describing the harness. The fourth is about safety: refusing in a chat does not guarantee refusing in action, and that has to be tested where the robot actually moves.

What it means

On narrow simulated suites, a general model with a harness already competes with trained policies and sometimes beats them — though at times with hints the policies never got. On broad tasks that require exploring the environment, success is still low: 19% on RobotWorld, 23.2% on RoboQuest, 16.8% on HumanCLAW. On the last one, the best model found the target in 64.9% of episodes but finished the task in only 16.8%: finding an object is not the same as dealing with it. The weak spots are physical and evidential: precise manipulation, locomotion (Astra solved none of the 11 locomotion tasks in RobotWorld), understanding its own body, and admitting that an answer cannot be known.

The ladder of delegable work
responsibility carried alone ↑
4Role
Run the household's daily breakfast
days and longer
3Workflow
Prepare breakfast for four, all ready on time
hours
2Task
Make a cup of coffee and bring it to the user
minutes
1Primitive
Reach a handle, pick up a cup
sub-second to seconds
What is entrusted
One operation → one task → several tasks → a standing role

The ladder of delegable work: progress is measured by the responsibility a robot can be trusted to carry alone.

So the paper proposes to measure progress not by the number of solved tasks but by the responsibility a robot can be trusted with. Robots already execute single actions and complete whole tasks, they are only beginning to reach workflows of several tasks, and no system in an everyday, unprepared environment has yet been reported to hold a standing role — say, running a household's daily breakfast — without human help. To climb higher, a robot needs more than intelligence: it has to be on time, stop when told, and prove that its action really changed the world the way it was asked to.

AI reviews in simple way

Every day we read fresh AI papers and retell the essentials in plain human language — no hype, no fluff. If you want to see where AI agents are heading before everyone else, subscribe.

New reviews — every day

Follow on X