8.3 billion simulated users, and the model playing them changes the verdict

When the average user hides what your product is doing

Almost every evaluation of AI systems and digital products suffers from the same disease. It tests an averaged abstraction — as if everyone had the same level of experience, the same conversational style, the same tolerance for errors, latency and strange answers.

In practice that distorts the picture. A beginner working through a coding task wants explanations and reassurance. An experienced developer wants a short answer and more autonomy from the agent. One user leaves after a chatbot's first hallucination. Another gives it a second chance. Measure all of it with a single number and none of that shows up.

MatrAIx: Simulating the World with 8.3 Billion Persona Agents proposes a different route: test products against a large population of simulated users who have different biographies, habits, values, skills and constraints. Instead of one typical user, a whole world of personas.

If you are building a chatbot, a website, an app or an AI assistant, knowing whether the system solves the task is not enough. You need to know who exactly it solves the task well for, and who it fails.

What MatrAIx is

MatrAIx is a large pipeline for evaluating products with simulated users. It has three main parts.

The MatrAIx pipeline: how a cohort is assembled from a large persona database, run through tasks, and reported back by subgroup.

The first is Persona 8B, a database of 8.3 billion personas. Each one is described by 1,290 categorical attributes. Some are basic: age, region, language, education. Others are finer-grained: conversational style, risk appetite, interests, health, digital habits, attitude toward AI, experience with particular tools, and so on.

The second is the execution environments. There are four:

🟠 survey — the persona answers questions and questionnaires

🟠 chatbot — the persona talks to an AI system

🟠 web — the persona browses websites

🟠 app — the persona uses a desktop or mobile application

The third is the task library, 1,010 tasks in the paper. They are scenarios for retail, finance, medicine, software and a couple of dozen other domains. How a user reacts to a price increase, for instance, or whether they keep talking after a chatbot error, whether they can find the feature they need in an app, how much response latency they will tolerate.

In short:

🟣 8.3 billion personas in the full population

🟣 nearly 1 million personas in the public release

🟣 1,290 attributes per persona

🟣 4 evaluation environments

🟣 1,010 tasks

🟣 18,189 runs in the main experiments

Where these people came from

This is the most interesting engineering in the paper. The authors did not simply generate billions of random profiles. They worked to preserve the dependencies between attributes.

Sample every field independently and you get nonsense. Language, region and English proficiency start combining absurdly. Age stops lining up with education and career. The profile looks plausible field by field and wrong as a whole.

To avoid that, the authors build a directed acyclic graph of dependencies. Every attribute has parents. English proficiency depends on native language and region. Education depends on age. Skills depend on career, environment and other attributes. The persona is then assembled step by step, in topological order.

The dependency graph over persona attributes: demographics and education pull skills, interests and behavior along with them.

This does not make the simulation real people, but it stops the system from mass-producing internally contradictory profiles.

Alongside the synthetic personas there are personas grounded in real sources. They were built from six kinds of data: Wikipedia biographies, Amazon review histories, the Stack Overflow developer survey, the General Social Survey, PRISM profiles, and a voluntary survey of the MatrAIx team itself.

The public release contains roughly a million personas:

🟠 599,847 personas grounded in real sources

🟠 400,000 synthetic personas

Direct identifiers are stripped. Names, contact details and other explicit links are not published. Only the extracted attributes and text descriptions remain.

Why 1,290 attributes is not overkill

On paper the number looks comical. The logic is clear enough, though: if you want to test different groups of people, the description has to be detailed enough to tell them apart.

The three-level Persona 8B taxonomy: five top-level attribute groups, 16 subgroups and 55 categories.

All the attributes fall into five top-level blocks:

🟣 background: demographics, language, education, career

🟣 psychology: values, personality traits, views, motivation

🟣 capabilities: domain knowledge, skills, tools, programming

🟣 behavior and interaction: habits, working style, AI adoption

🟣 lifestyle: interests, culture, food, health, sport

None of this is decoration. The idea is that you can assemble a cohort by more than simple filters like age and region. You can select, say, people with low trust in AI, high price sensitivity, a middling technical level and a preference for short answers. Or the opposite: users who want detailed explanations and will sit through the latency.

For a product team, that is close to a substitute for a good research brief.

How they tested it

One of the central problems with every user simulator: the model may not follow the persona it was handed. It gets a profile — likes jokes, writes at length, avoids jargon — and then answers in its usual voice.

The authors test exactly this, separately: whether the AI agent stays in the persona it was assigned.

They took 10 behavioral attributes. Among them: politeness, sense of humor, verbosity, jargon use, fondness for emoji, code commenting style, variable name length and others. Each attribute was tested in all four environments — survey, chatbot, web, app — for 400 runs in total.

Persona adherence, broken down by attribute and by execution environment.

The result:

🟣 366 of 400 runs

🟣 91.5% of cases

🟣 the target behavior was either expressed or correctly suppressed

The details matter here.

🟠 it worked best in the survey, chatbot and web environments

🟠 worse in the app environment

🟠 it is easier to make a model express something than to make it withhold something

🟠 politeness is especially hard to suppress when the base model is well-mannered by default

That last point is telling. When the persona is supposed to be blunt and rude, the model often slides back into a safe, smoothed-over tone. The persona competes with the model's own default settings, and does not always win.

That is a limitation. The user simulator here depends on which LLM is playing the persona.

What the applied tasks showed

Now the useful part: does the simulator help you see the differences between groups of users.

The authors ran 18,189 trials across eight tasks, using three models as the persona agents: Claude Opus 4.8, GPT 5.5 and Claude Haiku 4.5.

One example is a meal-planning chatbot. Personas talked to a fixed assistant and rated whether they would follow the plan it proposed. What is interesting is not only the final number but the dialogue trajectories themselves: who asks for substitutions, who argues about cost, who asks about portions, who spends longer spelling out restrictions.

Dialogue transition graphs for meal planning: conversations unfold differently across groups with different economic motivations.

The gap between groups shows up not only in the final answer but in how they get to that answer. More price-sensitive users reject suggestions as unrealistic more often and ask for substitutions. Wealthier users spend more of the conversation on the format of the answer and on eating-out options.

That is a step up from an ordinary benchmark, where all you would see is "answer generated" or "task solved." Here you can see where the friction starts.

There is an uncomfortable finding too. On several tasks the outcome depended heavily on which model was playing the user.

The short version:

🟣 on the price-increase task, the share of wavering users was 98.3% with GPT 5.5, 27.0% with Opus and 83.3% with Haiku

🟣 on the paid-plan-on-a-website task: 75.8%, 23.2% and 93.9% respectively

🟣 on whether users keep using a chatbot after an error, the differences were just as enormous

Same persona cohort, same product, same scenario — and the conclusions about the product can come out opposite.

The main practical takeaway of the paper: the model playing the user becomes part of the experimental conditions itself. It cannot be treated as a neutral shell.

Where this helps and where it stops

MatrAIx does not promise digital twins of humanity. It is not a replacement for field research, not a replacement for real interviews, and not a basis for decisions in sensitive domains.

The authors say so plainly: simulations like this are suited to pre-production checks, stress tests, subgroup analysis and comparisons between product versions. But if you want to draw conclusions about real people, you still need to validate with real people afterward.

Simulated users come with several hard limits:

🟠 they can flatten the differences between groups

🟠 they can amplify stereotypes

🟠 they are not always convincing at withholding information, refusing, or walking out of a conversation

🟠 the base model has a strong effect on the result

🟠 if the user model and the system under test are built alike, a self-approval effect is possible

That last risk matters most. If the same model family plays both the user and the system, you can end up with an inflated score simply because the model likes answers that resemble its own.

The bottom line

MatrAIx moves the evaluation of AI systems and digital products away from the abstract average user and toward a population with different biographies and habits. Instead of one final score, you get a breakdown by cohort, subgroup and behavioral scenario.

The paper's central idea: what needs measuring is not only the system's ability to solve the task, but how differently that same system lands with different users.

A few things worth carrying away from the work:

🟣 user simulation is already good enough for early product checks

🟣 large persona populations are useful when you are hunting for rare cases and differences between subgroups

🟣 persona adherence can be pushed to roughly 91.5%, which still does not settle the question of plausibility

🟣 the result depends on the persona model so heavily that it has to be named in the report every time

🟣 systems like this do not replace real user research; they narrow the hypothesis space going into it

If you are building an AI product, there is a clear practical thought here. Before you release a system into the world, it is worth running it past a set of different user roles rather than a single benchmark or a single demo. Not to arrive at the final truth, but to see in advance where the product irritates people, loses their trust, or misses what specific groups expected.

That is the point at which simulated users become a working tool.