Giving planning tokens extra credit beats GRPO on math reasoning

Reasoning tasks are a sore spot for many AI systems, even ones with solid factual knowledge. A new paper shows that reinforcement learning (RL) does more than push accuracy up — it rebuilds the model's internal logic into a hierarchy that runs from low-level execution to high-level planning. That explains where those aha moments come from. More usefully, it explains why the standard algorithms spend their effort in the wrong place, and how to fix that.

How to tell planning from execution

The authors introduce a simple but useful proxy: Strategic Grams (SG), recurring 3–5-grams that steer the course of a solution. These are patterns like “let’s try”, “we can use”, “but the problem mentions” and so on. Tokens inside such n-grams count as planning tokens; everything else is execution — computing, substituting, applying formulas. The SG set is collected from successful solutions and annotated half automatically, half by hand. What comes out is a functional split by role that doesn't hang on surface features.

What happens during training

The model first drills execution until it is automatic: perplexity and entropy on those tokens drop sharply — confidence goes up, errors go down. This stage is fast, especially for already strong LLMs, where the lower level is reasonably well tuned to begin with.

Then the learning front shifts to strategy: the variety of strategic moves grows, reasoning chains get longer and better ordered, and accuracy keeps climbing — now on the strength of planning. This is where the aha moments are born: the model finds new tactics and picks up branching, backtracking and reflection.

Why the usual metrics cover the tracks

Token entropy falls because execution tokens make up the bulk of the text, and RL leaves the model more confident on them. You could read that as exploration of strategies dying out — and be wrong. A far better place to look is semantic entropy over the strategic n-grams: it rises as the model widens its repertoire of plans, and it correlates better with final accuracy.

If strategy is the bottleneck, spreading the gradient evenly across every token — the way GRPO does — is the wrong move. The paper proposes HICRA (HIerarchy-Aware Credit Assignment), a layer on top of GRPO that amplifies positive credit and softens the penalty specifically on planning tokens. Formally, it scales the advantage at strategic positions by a coefficient of α=0.2. The policy shifts into the subspace of strategies, which raises the chance of discovering good plans and locking them in.

What the experiments showed

The tests ran on Qwen2.5-7B, Qwen3-4B, Llama‑3.1‑8B and MiMO‑VL‑7B, from base to instruct variants; the benchmarks include AIME24/25, Math500, AMC23, Minerva, Olympiad and others. The picture repeats: HICRA consistently beats GRPO.

On Qwen3‑4B‑Instruct the gains show up even from a strong baseline: AIME24 goes from 63.1 to 65.9, AIME25 from 58.8 to 62.1, AMC23 from 76.8 to 82.5, Olympiad from 55.6 to 59.7. On Qwen3‑4B‑Base the additions on Math500 are generous: 83 to 89. On Qwen2.5‑7B‑Base HICRA wins over GRPO on every task, and on AMC23 especially: 8.4 points. In the multimodal setup Pass@K saturates quickly, but semantic entropy still separates the methods — and HICRA is ahead again.

What RL actually affects

Annotating the errors shows that strategic misses disappear fastest under RL, not procedural ones. Which is logical: a wrong plan cancels out flawless execution. So the right objective should push the model toward finding strategies and cementing them.

Where the method stalls, and what comes next

HICRA needs a baseline of reliable execution. On Llama‑3.1‑Instruct, forcing strategy without solid low-level skill leads to instability and doesn't overtake GRPO. The conclusion is simple: you need an adaptive regime that knows where the current bottleneck is — firm up the procedures first, then amplify planning.

Why this matters

The idea of a hierarchy and a split by role carries well beyond math. Code, scientific reasoning, planning a long piece of text, a multi-agent system with divided functions — anywhere strategy and execution can be told apart, it pays to amplify the strategic tokens and to measure progress through semantic entropy. That makes training more predictable and saves attempts.