Hundreds of thousands of professional tasks helped train AI agents.

AI agents need more than working files

Imagine asking an AI agent to compare vendor proposals and prepare a spreadsheet with the calculations. The files are there, and the prices are listed—but one proposal doesn’t say whether delivery is included. The agent is left to guess or skip the calculation altogether.

For a person, that’s a minor issue that’s easy to fix. For AI-agent training, it’s a problem: when the teacher model gets incomplete or contradictory information, its answers become poor examples for the models it trains.

The authors of the WorkFactory paper propose designing work tasks with all the necessary inputs, but without giving away the answers. A separate model then checks each task, looking for anything that could prevent it from being completed, and helps fix the materials. The researchers used these tasks to train the WorkPilot family of models. On the GDPval-AA benchmark, one model’s score rose from 879 to 1,565 Elo.

WorkPilot-35B-A3B outperforms its base model on four benchmarks of professional tasks.

How to prepare a work task

A typical training task often boils down to an instruction and a correct answer. Professional work is more complicated. To prepare a spreadsheet, report, or recommendation, an agent may need information from several files, calculation rules, and details about the task. The agent should still do the intermediate calculations itself.

WorkFactory starts with the desired output and works backward: it identifies the calculations and decisions behind it, then determines what inputs they require. The system records these relationships in an information-support graph. The graph shows what to include in the prompt and reference files, where to put the necessary information, and which results the agent should produce on its own.

In the paper’s example, the task is to choose a supplier of cardboard boxes. The agent gets a demand forecast, prices, delivery terms, and discounting rules. It must calculate the costs and net present value of each proposal, then make a recommendation. If the source materials omit one supplier’s delivery cost, the checking system can spot the gap. The missing term can then be added to a file, while the calculations remain the agent’s responsibility.

WorkFactory plans the data a task requires, checks the collected materials, and fixes any obstacles before generating work examples.

The key principle is to give the agent the basis for an answer without handing it the answer itself. If the files already include a comparison of the suppliers, the task gets easier—but it no longer tests the intended skill. Leave out an important term, on the other hand, and the result will depend on guesswork.

Tasks are assembled from three sources: job descriptions, information about workplace skills, and real professional documents. This gives the system a variety of tasks, from data analysis to preparing files for a specific audience. A language model then checks candidate tasks for coherence, feasibility, and plausibility in a workplace context.

The materials then go through a multi-step review:

🟠 A separate model attempts the task using only the files and information that the eventual agent will receive.

🟠 It looks for obstacles, such as missing data or contradictions between files, and distinguishes them from its own mistakes while solving the task.

🟠 If the task can’t be completed, the materials are revised and checked again. Tasks that still can’t be made workable are discarded.

The teacher model is allowed to attempt a task only after it passes this review. That reduces the risk of training examples containing answers based on invented assumptions or incomplete information.

What happened at scale

The researchers assembled 112,686 tasks across 612 professions. Each task contained an average of 4.15 reference files. The teacher model took about 57 steps on average to complete a task and made roughly 63 tool calls. The researchers used its outputs to fine-tune three Qwen-based models with 4, 27, and 35 billion parameters.

All three WorkPilot models outperformed their respective base models on four professional-work benchmarks. For example, WorkPilot-35B-A3B rose from 879 to 1,565 Elo on GDPval-AA, while WorkPilot-27B scored 1,612 in a local evaluation. That put it in the range of leading models compared in the study.

The gains showed up on other tests, too. WorkPilot-35B-A3B improved from 28.91 to 62.35 on JobBench and from 7.14 to 21.18 on OmegaUse-OfficeVal. That suggests the training helped beyond a single evaluation.

The researchers also tested what happens as the number of training tasks grows. For the 35-billion-parameter model, the first 5,000 tasks added 447 Elo points. Expanding the dataset from 75,000 to 110,000 tasks added another 13. More data helps, but the returns diminish over time. And how quickly they diminish depends on the model’s size.

The agent’s behavior changed, too. After the initial stages of training, it tackled longer tasks and used tools more often. As the dataset grew further, its trajectories became shorter on average, while performance continued to improve. The share of runs that hit the 250-step limit fell from 21.9% after training on 2,000 tasks to 2.7% after training on 110,000. Code execution failures also became less common. Web-search errors declined, but page-loading errors remained frequent.

What planning and checking improved

The authors compared three ways of preparing tasks: conventional file planning, planning with an information-support graph, and the same graph-based approach with additional checking and repair. Each method used a set of 5,000 tasks.

The graph reduced the share of tasks with missing data, incomplete instructions, and broken calculations. For example, errors in calculation chains appeared in 24.86% of tasks with conventional file planning, compared with 6.86% when using the graph. But dependency maps alone weren’t enough: some errors, including contradictions and answer leakage, actually became more common.

Execution-based checks added another layer to the planning. After repairs, the share of tasks with inconsistent information across files fell from 25.74% to 4.84%. Problems that prevented task completion dropped from 6.64% to 0.48%. A model trained on the repaired tasks scored 1,326 Elo, compared with 1,245 for the model trained with conventional planning.

The more tasks used for training, the higher the score on GDPval-AA, though the gains gradually tapered off.

Execution-based checks require additional compute. In the main dataset, about 70% of tasks needed no repairs, around 20% went through one round, and 2.48% reached the maximum of three rounds. Tasks that remained impossible to complete after that were removed.

There’s another wrinkle: results depend on which model serves as the teacher. For one set of 30,000 tasks, the researchers collected examples from two teacher models. A student trained on the first model’s examples scored 1,483 Elo; one trained on the second model’s examples scored 1,248. Both versions beat the base model, but the gap shows that task quality and dataset size aren’t the only factors. It also matters how the teacher carries out the work.

Conclusion

WorkFactory offers a way to prepare training data for AI agents that goes beyond simply adding more workplace scenarios. The system first identifies what information the task requires, then checks whether the available materials are enough to complete it, and fixes any gaps it finds.

The results show that this approach can improve models across several professional benchmarks, and that checking tasks makes training examples more useful. But there are limits: the teacher model worked only with text, so the study doesn’t test training on images or other visual data. All the models were fine-tuned on completed-task examples; the authors leave reinforcement learning for future work.

For AI agents tasked with handling files, calculations, and tools, coming up with a plausible task isn’t enough. You also have to make sure it’s doable—and that the teacher’s example demonstrates the skill the model is meant to learn.