Model reasoning traces break into the same episodes human solvers use

Large reasoning models (LRMs) today don't just answer — they unfold long chains of thought. That lets them handle harder problems, but it creates a new one: how do you describe the structure of that reasoning, and how close does it come to human thinking? The researchers propose borrowing a framework well tested in cognitive science — Schoenfeld's episode theory, built originally to analyze how people solve non-routine math problems. The idea is simple: annotate the text of a model's solution as a sequence of episodes — Read, Analyze, Plan, Implement, Explore, Verify, Monitor — and see which transitions come up most often and how they relate to getting the answer right.

Where the episodes come from

Schoenfeld showed that a skilled problem solver doesn't move at random but in episodes: reads the problem, breaks it down into its key relationships, plans the steps, carries out the computation, occasionally tries a detour, checks the work, and monitors the process throughout. Modern LRMs like DeepSeek-R1 or GPT-o1 also write at length, with pauses and markers of attention, which makes their output convenient to annotate this way. The paper finds that Read→Analyze and Plan→Implement transitions are as consistent in models as they are in people, and that Explore and Verify kick in when a solution needs a second look.

How the corpus was built and labeled

The researchers took a bank of SAT math problems with detailed metadata and generated solutions with the open LRM DeepSeek-R1. Then came two levels of annotation: paragraphs are tagged General, Explore or Verify (to capture the overall approach), and every sentence gets one of the seven episode labels. The labeling followed purpose-written guidelines, went through a pilot, and used trained annotators, producing the first open corpus for analyzing machine reasoning. The result is thousands of labeled sentences and paragraphs plus the full instruction package for reproducibility.

What the analysis shows

The map of transitions between episodes makes sense. Most often the model moves from reading to analysis and from planning to execution, while Explore loops back to Analyze to sharpen its understanding. That resembles the patterns of human experts and suggests models already develop a governable structure for working through a problem.

The authors then ask whether LLMs can label these episodes automatically. They compared GPT-4.1, GPT-4o and Gemini-2.0-flash across several setups: plain zero-shot, with examples, with a detailed guide, and with both together. The headline result is that a good instruction measurably improves accuracy. For GPT-4.1 at the paragraph level, accuracy rose from 0.444 to 0.740 with the instruction, and to 0.757 when the instruction was combined with examples; at the sentence level it reached 0.681. Gemini gains from instructions too. The takeaway: a standard prompt is a poor fit for cognitive annotation, while carefully written rules produce a jump in quality.

The authors also trained conventional models on the labeled data: BERT, RoBERTa and simple classifiers over embeddings. The best trained system falls short of instruction-guided GPT-4.1 but comes close: BERT reaches about 0.73 accuracy on the test set. The errors are telling — Analyze gets confused with Verify, and Implement with Verify. The model takes a computation step and checks it in the same breath, and neither people nor algorithms always find it easy to draw a clean line.

Why this matters

The work delivers more than a label set. It's a theoretically grounded scheme that ties a model's outward "verbalization" to legible cognitive states. That matters in several directions:

• interpretability: you can see where the model is planning, where it is wandering, and where it is checking itself;
• control: prompts or policies can reinforce the episodes you want — making it verify more often, for instance;
• training: correct transitions can be rewarded and pointless Explore loops penalized.

The corpus is built on SAT problems, which sit at a moderate level of difficulty. Olympiad math and other domains are still ahead. Automatic annotation already works, but there is distance to cover before it is stable in production settings. And the borderline cases between episodes will need the rules tuned further, and possibly richer context.

Bottom line

The authors carefully port Schoenfeld's theory into the world of LRMs and show that long model reasoning really does fall into episodes with recognizable transitions. The open corpus and the detailed guidelines set a standard for future work on interpretability and on more controllable reasoning systems.

💾 Code