Long programming tasks require a model to do more than write code: it has to assemble many pieces into a working project. RepoFactory automatically turns real repositories into training tasks, then improves them using low-cost checks. Training on these tasks significantly boosts the model’s performance on whole-project development.
The main idea
Training environments are usually checked for basic usability: does the code run, and do the tests pass? RepoFactory adds a second step: it changes the tasks to involve more work and more connections between components. The authors then test whether training on these tasks helps on independent benchmarks.
🟠 A real repository becomes a task: the agent must recreate the project from a description without seeing the original code.
🟠 The environment is scored on three criteria: whether it passes validation, how closely its components interact, and how much code it requires.
🟠 The environment is checked again after every change; unsuccessful versions are rolled back.
🟠 A model trained on these tasks significantly improved on repository creation and other software-agent tasks.
What is RepoFactory?
RepoFactory is a pipeline that turns source projects into testable training environments. It finds repositories, creates an isolated environment for each one, writes a task description, and then improves the task based on automated checks.
RepoFactory finds repositories, builds environments for them, creates tasks, and improves those tasks based on validation results.
The system has several connected parts:
🟣 Project discovery. The system selects open-source repositories with package metadata and tests. Projects that require external services, network access, or specialized hardware are excluded: the environment must run in isolation.
🟣 Environment setup. A container is created for each project—a separate runtime environment with its dependencies. The system verifies that the tests pass in a clean container with no network access.
🟣 Task creation. An analyzer extracts public functions and classes from the code, along with the interfaces called by the tests. An agent uses this information to write a description of what needs to be implemented. The original project and test code remain hidden from the agent being trained.
🟣 Environment improvement loop. The system checks the description, implementation, and tests, then revises them together. This can fix a flawed environment or increase the amount of work required by the task.
Three scores guide these improvements. Validity means that the original implementation passes the tests while an empty environment does not. Connectivity measures how closely the parts of the program interact. Implementation scale accounts for the amount of code and the number of dependencies. The scores are multiplied together: if any one of them is zero, the overall score is zero too.
These are rough indicators of training value, not direct measures of how much an agent will learn. That’s why the authors separately test the effect on a trained model.
How project reconstruction from a description works
Suppose RepoFactory selects a data-processing library. It needs to turn it into a task where an agent recreates the library without access to the original project.
🟠 First, the system runs the library in a container and tests it. If dependencies fail to install or the tests don’t work, the environment is fixed. If that fails, the project does not pass the validity check.
🟠 Next, the analyzer determines which functions and classes the test suite uses. Based on that information, an agent writes a project description and a list of interfaces to implement.
🟠 The system compares the description with the list of symbols called by the tests. If a required interface is missing, the check reports an error and the description is revised.
🟠 The system then checks that the original library gets a full score and an empty folder gets zero. If either condition isn’t met, the agent reviews the report and changes the description, code, or tests. If the task is too small or its components barely interact, the agent tries to expand the project or strengthen the connections between its components.
🟠 Every new version is checked again. If the selected score hasn’t improved, the system reverts to the previous version. The loop can run up to 12 times.
The number of environments that pass the validity check grows as issues are fixed.
The agent doesn’t solve the entire task during this process. Instead, RepoFactory uses test reports and code analysis—cheaper than repeatedly launching a lengthy solution attempt and waiting for the result. But these checks can miss things. For example, a description may become complete in a formal sense while the task still requires no meaningful programming.
How RepoFactory is evaluated
The authors assess the system in two ways: whether it can create more valid environments, and whether training on them improves the model.
They use five benchmarks to evaluate the model. NL2Repo measures whether it can create a complete repository from a description. Three versions of SWE-bench test whether it can fix bugs in existing projects, including projects written in different languages. Terminal-Bench evaluates tasks performed in the command line.
Comparing RepoFactory with a version that has no improvement loop reveals the contribution of refining the environments. To assess the value of the data, the authors also train models on other programming-task datasets. The comparisons use the same model, a similar number of training examples, and the same evaluation setup. Previously published results from other studies, however, were obtained under different conditions.
One experiment tests whether increasing the number of environments helps. Another compares training on environments with different levels of complexity and connectivity. This helps the authors distinguish the effect of scale from the effect of the improvement loop itself.
Accuracy on the NL2Repo benchmark rises with the number of training environments.
The numbers
Training improved performance on all five benchmarks, but the gains varied: repository creation saw the biggest jump. The version with the environment improvement loop also outperformed training without it: NL2Repo scores were 46.4% versus 41.3%, and the average across the five benchmarks was 53.5% versus 51.5%. The gains came not just from the number of tasks; automatically refining them also improved results.
Conclusion
RepoFactory shows that real repositories can be turned into testable tasks that require building a whole project, not just fixing an isolated piece of code. A model trained on these tasks improved not only at creating repositories, but also at fixing bugs and working in the command line.
The main limitations are coverage and the accuracy of the scores. So far, the system focuses on Python projects, and its three quality metrics describe each task only in broad terms. They don’t directly identify which programming skills an agent will need. Future work will expand language support and more precisely assess what each environment teaches.
Read next
InclusionMed: Which AI should I choose for medical tasks?
Why Is It Difficult for an AI Agent to Act in the Real World?
WorldFoundry: How can you distinguish video generation from world
How AI Can Enhance an Individual’s Intelligence
FrontierPhysics: Let’s see whether AI agents can make discoveries
How AI Chooses the Right Model for Each Task
CaPtain: How to control a robot using a small AI model
WorkFactory: Hundreds of thousands of professional tasks helped train
How AI Agents Learn General Skills Across Different Tasks
How can you tell that an AI agent is doing the task incorrectly?
How an AI agent can master a new skill through practice using the Socratic method
How an AI agent improves interfaces on its own
AI reviews in simple way
Every day we read fresh AI papers and retell the essentials in plain human language — no hype, no fluff. If you want to see where AI agents are heading before everyone else, subscribe.
New reviews — every day
Follow on X