Long horizons alone can collapse RL training for LLM agents

There is a lot of noise around LLM agents right now: we teach models to use tools, browse websites, fix code, work through multi-step tasks. It looks as though the central question is the quality of the model itself, the size of the context window, or the cleverness of the training algorithm. But the authors of "On Training Large Language Models for Long-Horizon Tasks" offer a far more mundane and far more useful explanation: sometimes an agent fails not because it cannot reason, but because the goal is simply too far away.

Agents break down because the path is long, not because they are dumb

It is a subtle distinction, but a fundamental one. A model that knows the rules of Sudoku and can solve a puzzle in one shot will not necessarily solve the same puzzle reliably across 25–30 consecutive steps of interaction with an environment. Errors pile up along the way, the reward signal thins out, and reinforcement learning starts behaving less and less predictably.

This paper separates reasoning difficulty from the length of the path to the goal carefully, almost surgically — and shows that a long horizon is a serious bottleneck in its own right. For anyone building agent systems that is a consequential finding: before inventing ever more elaborate RL recipes, it may be worth first cutting the number of steps the agent is forced to take.

What the paper actually tested

The core idea is simple: work out how horizon length affects the training of LLM agents when everything else is held as constant as possible.

By "horizon" the authors mean not just a cap on steps but the actual length of the action chain needed to reach the goal. They separate several things:

The key move is to build tasks where the rules and the structure of the reasoning stay the same and essentially only the length of the solution changes.

For that the authors picked two convenient text worlds:

In the Sudoku case the authors deliberately filtered the puzzles so that only basic solving techniques are required. That keeps two causes of failure apart: "the model does not know how to solve this" and "the model knows how, but cannot hold a long sequence of actions together."

Another strong methodological move: the authors checked whether the model could solve the same instance in a compressed, short-horizon format — emitting the entire finished board at once instead of filling Sudoku in cell by cell. If the model handles that, the underlying solving ability is there, and whatever goes wrong afterwards can be pinned on the length of the interaction.

The main result: a long horizon breaks RL all by itself

The central finding reads almost like a diagnosis: increase only the length of the solution and reinforcement learning becomes unstable, and can collapse outright.

The authors trained Qwen3-1.7B, first with SFT on expert trajectories and then with RL on top. On short tasks everything looks familiar: RL helps, scores go up. As soon as the distance to the goal grows, the picture changes sharply — heavy oscillation first, then a collapse in performance.

What makes this valuable is that the authors do not stop at "long tasks are worse." They try to isolate the cause, and their conclusion is that the task has not necessarily become intellectually harder. Even when the reasoning is essentially the same, adding steps by itself makes training far more brittle.

Why does this happen? The paper points to two mechanisms.

First, exploration of the action space. The longer the chain, the smaller the chance of stumbling into a successful trajectory by accident or half-intent. A mistake near the start poisons everything that follows.

Second, credit assignment. When reward is sparse and arrives only at the end, a failure pushes a bad signal onto the whole chain — including intermediate steps that may have been correct. For a language model that is particularly nasty: a negative signal does not carefully nudge it toward the right answer, it spreads probability mass across an enormous vocabulary and amplifies noise.

The observation maps cleanly onto real systems. A coding agent may know how to fix an individual bug, but if success requires twenty dependent steps — reading logs, editing a file, running tests, fixing the errors that fix caused, checking again — training becomes far less reliable.

Horizon reduction turns out to be a surprisingly plain fix

The most practical takeaway: you do not necessarily fight a long horizon with cleverer RL, but with a structural cut in the number of steps.

The authors call this the horizon-reduction principle and propose two main ways to apply it:

This is where the paper is at its most convincing: the idea looks obvious, and the experimental effect is very large.

In Sudoku the agent was allowed to fill several cells per step rather than one. In Rush Hour it could make larger moves. The result: training is steadier, converges faster and, most importantly, does not fall apart on the long scenarios where the primitive-action baseline fails outright.

The authors checked separately whether the gain is a side effect of "a stronger policy." The ablation is elegant: a policy trained with macro-actions was made to act in the environment as though only single primitive steps were available. Stability degraded again. So the operative factor really is the reduction of the effective horizon, not the model simply having gotten smarter.

Separately interesting: not all macro-actions are equally useful. Rigidly fixed batches of steps work worse than flexible macro-actions whose length the policy picks for itself. That matches a sound engineering instinct: aggregation that is too coarse can hurt as much as no aggregation at all.

Subgoals work too, which is good news for real agents

The second route is decomposition into subgoals. In Sudoku the authors used intermediate, checkable targets such as correctly completing an individual block. That turns one long episode with sparse reward into a sequence of shorter segments with dense signal.

This matters beyond puzzles. In real systems we already build agents around intermediate checkpoints by instinct: find the relevant file, then localize the bug, then propose a fix, then run the tests. The paper gives that habit an empirical justification rather than a merely engineering one: subgoals shrink the effective horizon and make training easier.

In effect, the paper leads carefully to the thought that many successful agent architectures work not because their planning is especially elegant, but because they quietly shorten the horizon — through action hierarchies, API calls, code as a high-level action, intermediate checks and dense feedback.

The best part: horizon reduction helps even on longer tasks the model never saw

One of the most satisfying parts of the work is an observation the authors call horizon generalization. A model trained on tasks of moderate length can transfer decently to longer instances, as long as the reasoning difficulty stays the same.

It is an important result and, honestly, a slightly counterintuitive one. You would expect an agent that never trained on very long episodes to do badly on them. It turns out that if training was stable and per-step accuracy is good enough, the model can stretch further than anything it saw during training.

The macro-action variants transfer best, for two reasons:

In practice that means this: you do not always have to train an agent on the longest and most expensive scenarios from the start. It is often smarter to get it through shorter trajectories reliably first, then rely on transfer and a curriculum over task length.

The authors test that idea too. In Rush Hour they compare training directly on long instances against a schedule that starts short and then moves to long. The curriculum wins. That is one more argument that short-horizon competence is not a pleasant bonus but the foundation everything else is built on.

Why this paper matters beyond Sudoku

At first glance the work can look like "another RL-on-toy-environments paper." That would be a shallow reading.

Its real value is that it offers a simple axis for analyzing agent systems: before arguing about model size, optimizer tricks and prompt quality, ask what the effective horizon is here.

That question is useful across a lot of applied settings:

The authors also show that the effect is not confined to one task, one model or one optimizer. Similar patterns show up in WebShop, on a larger model and under a different style of optimization. That strengthens the central claim: horizon length is not a quirk of one implementation but a constraint that cuts across the whole class of LLM agent systems.

There is an important limitation as well. The paper is honest that horizon generalization does not mean generalization across reasoning techniques. Once Sudoku starts demanding fundamentally harder techniques, transfer breaks. That is a sober conclusion: RL is good at stretching abilities a model already has over longer trajectories, but it is under no obligation to produce qualitatively new ones.

The takeaway

The value of this paper is not in headline numbers but in how precisely it names the problem. For LLM agents the bottleneck is often not the reasoning itself but the length of the path to the goal. And if that is right, the best next move is not necessarily to complicate RL but to shorten the effective horizon.

What follows is close to an engineering manifesto:

For anyone building agent systems — from web automation to developer tooling — this is a very practical lesson. Sometimes the way to a stronger agent runs not through a bigger brain but through a shorter path.