Microsoft builds synthetic companies to test Copilot

From test prompts to simulated workplaces

A useful enterprise AI evaluation has to test more than whether a model can answer a generic question. It must also test whether Copilot can find relevant information across an organization, summarize it accurately and preserve details that matter to a particular industry.

Source: microsoft.com

Microsoft’s platform builds synthetic workspaces with users, meetings, emails, documents and organizational structures. Each workspace, which Microsoft calls a “tenant,” is an isolated environment that teams can adapt for their own evaluation needs.

To make those environments plausible, Microsoft drew on public sources:

The environments also model change over time: meeting histories, project schedules, email threads that unfold over weeks and shifts in organizational structure. That lets teams test questions about what happened at a particular point or what changed after a review, rather than only asking about a static snapshot.

Microsoft created three industry environments:

A pipeline for evaluation prompts

The synthetic workspaces provide the material for testing, but teams also need realistic questions grounded in that material. Microsoft’s automated pipeline starts with a library of common work tasks, selects those relevant to an industry, adapts them to its terminology and context, then generates and checks prompts against the synthetic environment.

Figure 1: Components of Industry Evaluation Data Generation Platform

Source: microsoft.com

Microsoft also evaluates whether the generated data are fit for purpose. Its internal semantic evaluation system checks organizational structure, contextual relevance, behavioral realism and content safety, including whether material contains harmful or personally identifying content or appears to represent real people.

The platform’s practical case is partly about repeatability. Microsoft says automation replaced preparation that took days with a pipeline run, allowing teams to recreate an evaluation set from the same configuration and get an identical result. The company estimates savings of about $100,000 per evaluation set and $300,000 across the project.

What the reported gains establish

Since the project began, Microsoft reports a 39% improvement in a composite measure of how closely synthetic data resemble real corporate environments in complexity and patterns. Some industry environments scored above 0.9 out of 1.0. Each environment now includes hundreds of users and a complete communication history.

Those figures make a case for better synthetic data and faster evaluation. They do not, by themselves, show that Copilot became more reliable for customers. The announcement describes a platform for testing and reports improvements in the data used to test; it does not give results for Copilot’s accuracy, omissions or performance against a specific benchmark.

I think that distinction matters. A realistic test environment is useful only if its evaluations predict failures in real organizations. Microsoft says the data are checked for structural, contextual, behavioral and safety qualities, but the account does not say how well those checks correlate with real-world outcomes.

The privacy design is central to the proposal: Microsoft says no real customer data enters any stage of the pipeline. It also cites privacy-preserving aggregation methods, legal review of public sources and provenance tracking throughout the process. That makes the datasets easier for teams to share and reuse, while leaving the harder question of evaluation validity open.

Microsoft plans to extend the work to multi-step interactions with AI agents, organizations that change over months or years, and systems that take actions rather than only generate text. That shift will put more weight on the quality of the test environment: when AI moves from drafting answers to acting across workplace systems, a convincing simulation will need to test not just what it says, but what it does.

The industry data evaluation sets were developed by Somya Chatterjee, Harshita Sahijwani, Min Chen, Sushmita Rajeswari Muppa, Daequan Julien, Irene Schaffer, Chandra Inguva, Jenny Chen, Minha Hwang and Sam A. Jacobs.