How an AI agent can master a new skill through practice using the Socratic method

A Skill Needs to Be Tested

One successful run doesn’t mean an AI agent has mastered a useful procedure. It may have solved the task by chance, missed important conditions, and written a skill that only works in one situation.

In their paper “Socratic Skill Learning,” the authors propose a different approach: an expert guides the agent as it works through a task. The agent uses tools and writes the skill itself, while the expert asks questions, suggests checks, and points out gaps. The agent can test, revise, or reject the advice.

A skill should be based on verified actions and the conditions under which they work. So the agent doesn’t just receive a ready-made instruction. It figures out when the instruction applies and decides what to keep.

What Is Skill Learning?

The method is called Socratic skill learning. It involves two AI agents:

🟠 The learner works on the task, runs tools, tests hypotheses, and updates the skill package.

🟠 The mentor reviews the learner’s work and the current skill package, but cannot perform the task or edit files.

🟠 After each training round, a separate, fresh model instance tests the skill on the original task. At the end, the skill is tested on new tasks from the same family.

That last step is essential. If the learner has simply memorized the answer to the training task, a fresh test won’t reveal it. For a skill to count as transferable, it has to help with new versions of the task.

The researchers tested two schedules. In the first, the mentor checks in at regular intervals—for example, after every ten learner actions. In the second, the mentor returns when an unresolved question arises or the data changes in a way that may call for new advice. These questions are stored along with evidence and changes to the skill. If an earlier solution stops working, the question can be reopened.

The learner investigates the task and writes the skill, while the mentor guides the checks and returns to unresolved questions.

Testing on New Tasks

For their experiments, the authors assembled a skill-transfer benchmark: 408 executable tasks grouped into 35 families. The tasks span six areas, including systems programming, data processing, finance, and office automation. Each family includes an initial training task and other tasks for testing transfer.

The description for each family specifies what stays the same and what can vary. For example, database recovery tasks may differ in log format, data integrity, and recovery depth. The learner sees this description and investigates the task with those variations in mind, but can’t access the test tasks until training is over.

The main comparison covered 22 families and 155 test tasks. For each skill, the researchers launched a fresh model session for every test. The score reflected the share of requirements the solver met.

Skills developed through periodic check-ins scored 0.801. That was higher than the score for a no-skill method (0.723), an adapted method for extracting rules from work sequences (0.761), and a method that compares the work of a teacher and a learner (0.751).

The new scheduling approach performed just as well. Training around unresolved questions scored 0.805. That’s only a small gain over periodic check-ins, but it used 61.7% fewer mentor tokens. Total tokens for training and testing fell by 16.9%.

Comparison results: a score of 0.805 with unresolved-question training; 61.7% fewer mentor tokens; 16.9% fewer tokens overall.

Study overview: task families, training approaches, and comparison of skill quality and mentor costs.

Why the Mentor Needs to Show Up at the Right Time

The authors compared not only schedules, but also who creates the skill and when guidance arrives. Across several model pairs, in-the-moment mentoring outperformed having an expert create the skill or offer advice after the task was complete.

Timing matters. While the learner is investigating a task, advice can influence what it checks next. Wait until the end, and some important decisions have already been made—and unverified assumptions may have made their way into the skill.

After a lesson, the learner was more likely to write or edit files. The chance of taking one of those actions on the next step rose by 23.9 percentage points. The chance of changing the skill over the next ten actions rose by 38.6 points. This observation doesn’t prove that the lesson caused every change: the mentor’s timing depended on how the work was progressing. But it shows how advice shaped the process by prompting new checks and revisions.

A financial-reporting example makes the point. The mentor suggested looking for discrepancies by comparing figures within the report. The learner first wrote a general check and flagged four suspicious figures, even though there was only one error. It then examined the results and added financial dependencies and reverse calculations. The revised check identified the correct row. The skill retained more than the mentor’s words: it included a refined procedure and examples of the conditions under which the procedure could be trusted.

As performance on the original task improves, the mentor gives fewer new suggestions, but the learner continues to update the skill.

When a Fixed Schedule Costs More

Early in training, the mentor’s advice is often useful: the learner doesn’t yet know what to check. Later, the need for guidance declines. In the periodic-check-in experiments, the share of reviews that led to new advice fell from 73% in the first round to 16.1% in the third.

But work on the skill didn’t stop. After successfully completing the original task, the learner updated the package in 98 of the next 99 rounds. It kept refining the instructions even when no new advice was needed.

That makes a fixed schedule inefficient. By the final round, reviews used 5.3 times as many tokens as they had at the start, because the mentor had to process all the accumulated context each time. And 85.8% of review costs in the final round went to checks that produced no new advice.

Training around unresolved questions cuts down on unnecessary check-ins. The mentor receives information about the current question, recent changes, and the learner’s latest work. If a question has been resolved, there’s no need to review it again. If a fresh model fails on the original task, the related question is reopened.

Comparison of the costs of skill authoring, mentoring, and result validation across different training schedules.

What the Results Show

On the main task set, training around unresolved questions maintained skill quality while reducing mentor costs. The researchers also repeated the comparison with other model pairs on the benchmark’s shared subset. The new method used 63.8% fewer mentor tokens for one pair and 77.4% fewer for another, while keeping scores within one percentage point of periodic training.

Lower mentor costs don’t always mean the entire process is cheaper. In an experiment with one of the models, the learner took more actions, increasing the total token count. The method consistently reduces the cost of external guidance, but overall costs depend on how much extra work the learner takes on.

There’s another limitation: the experiments center on one initial task per family. Additional tests show that the skill transfers to new cases within the same family, but don’t tell us how well it will work in a different domain. The study also compares complete protocols, rather than isolating the contribution of the mentor from that of the schedule or the unresolved-question log.

Conclusion

A transferable skill comes from testing a procedure under different conditions. In this approach, an expert helps the learner figure out what else needs checking, but the learner creates the skill. That turns advice into a practical rule, verified in use.

Guidance during a task may be more useful than a review afterward. Tracking unresolved questions helps teams spend fewer mentor resources once the learner no longer needs advice at every step. Teams building AI agents should store not just instructions alongside their skills, but also evidence of when those skills work and which questions remain open.