The AI model with the highest score on a medical benchmark won’t necessarily perform best on a particular clinic’s tasks. Rankings shift depending on whether you measure reasoning, actions, or work with medical records. InclusionMed brings these evaluations together in one framework and shows which tasks it covers—and which are still largely missing.
An overall score can look more conclusive than its foundations justify: it depends on which tasks are included and how they’re weighted.
The main idea
InclusionMed is an open framework for evaluating medical AI models. It links benchmark selection, reproducibility checks, and model comparisons, so you can view the overall ranking alongside results for specific roles and capabilities.
🟠 The Atlas maps medical tasks by their role in healthcare and the AI capability involved: knowledge, reasoning, or action.
🟠 Candidate benchmarks are assessed against six criteria, then independently reproduced. This filters out unsuitable tests and those that no longer distinguish between models.
🟠 The overall index averages results across five benchmarks. Separate scores show how models perform on different types of tasks and how much it costs to run them.
What makes up the InclusionMed system
The system starts with a coverage map and ends with a comparison of models. The Atlas is a matrix where three healthcare roles intersect with three AI capabilities. The role indicates whose work or needs a task represents; the capability indicates what the AI is expected to demonstrate.
The InclusionMed process: gather candidate benchmarks, audit and independently reproduce them, then evaluate models and calculate the overall index.
The framework has three main parts.
🟣 Collection and documentation. The team gathered benchmarks from research papers, open platforms, and community submissions. Large language models help turn information about the tasks, data, scoring, and availability into benchmark profiles, which people then review. The collection process is repeated to incorporate new tests and updates.
🟣 Audit and reproduction. Candidates are assessed for medical relevance, the reliability of their sources and scoring, availability, contribution to overall coverage, currency, and ability to distinguish between models. Researchers then run the same models on selected benchmarks and check whether they can reproduce the reported scores.
🟣 Five benchmarks and an index. The first release includes four open benchmarks and one closed benchmark developed by the authors. Each carries the same weight: 20%. The index is supplemented by scores for each role and capability, as well as an estimated cost per task.
The Atlas currently covers only three of the nine possible role-capability combinations. It has no benchmark focused primarily on helping patients and the general public, and none of the included tests represents knowledge as a distinct capability.
How it works: from benchmark checks to rankings
Imagine a new benchmark in which AI has to perform tasks for a healthcare organization. The system first evaluates the test itself, then compares models on it.
🟠 First, the benchmark is documented. Information about the tasks, data sources, scoring rules, and license is collected from the paper, website, and available materials. Incomplete documentation or unclear sources for the correct answers can be grounds for rejection.
🟠 Next, it is placed in the Atlas. The team determines whose work the tasks represent and which capability they test. If most tasks involve actions performed by administrative staff, the benchmark is classified under the healthcare administration role and the action capability.
🟠 Then come the audit and reproduction. Researchers assess the benchmark against six criteria and run the same set of eight models on it. A test is excluded if it barely distinguishes between models or the top scores hit a ceiling. The process can also stop here—for example, if the code is unavailable, the reference answers are questionable, or the scoring cannot be reproduced.
🟠 Finally, the models are evaluated and the index is calculated. A benchmark accepted into the Atlas becomes part of the comparison. Each model is scored on five benchmarks, and their equally weighted average forms its overall result. If you care specifically about administrative work or reasoning, you can look at the score for that category instead of relying only on the overall ranking.
The index’s composition affects the conclusions: a model may do well on clinical questions but struggle with procedural tasks. If there’s no suitable benchmark, the ranking won’t fill that gap.
How it’s tested
The first release compared 22 model versions from 11 families. The evaluation covers five benchmarks: responses to clinicians’ professional queries, work with electronic medical records, computational tasks in medical research, administrative workflows, and diagnostic reasoning.
Models were run according to each benchmark’s rules. In one case, answers were evaluated against clinical criteria; in another, a task counted as successful only if every checkpoint was passed; in a third, both the diagnosis and its justification had to match the reference answer. All scores were converted to a scale of zero to 100, but the underlying metrics differed. The overall index is a useful guide, but it does not make these tasks fully equivalent measures.
To assess the benchmarks themselves, eight models were tested on eight candidates. A benchmark was excluded if the three top-performing models scored at least 85, or if the spread between model scores was no more than four points. This helps identify ceiling effects and weak differentiation, though it doesn’t establish how well the tasks reflect real clinical work.
The authors also compared estimated inference costs—the cost of processing requests with a model. Estimates were based on the number of input and output tokens and published prices. Some costs for failed runs, retries, evaluation, and tools were excluded, so these figures do not represent the full cost of deployment.
Overall score versus estimated cost: models with similar results can come with very different costs.
The numbers
The figures show both the scale of the selection process and the limits of the comparison. Only a small fraction of the 388 candidates made it into the index, and model scores varied considerably. Similar overall scores don’t necessarily mean similar costs: choosing a model depends on the tasks you need it for and the cost of evaluating it regularly.
What it means
InclusionMed turns medical AI evaluation into a system where you can see not just the ranking, but what it’s based on: which roles and capabilities are represented, how tests were selected, and where the gaps remain. Individual tasks reveal differences that the overall score can smooth over. For example, the ten leading models performed better in patient communication than in clinical decision-making.
But the current index covers only three of the Atlas’s nine cells. Most tasks are in English, text-based, and set in US healthcare contexts; direct patient assistance is not represented. In addition, only 104 candidate benchmarks had enough materials for independent reproduction.
A ranking can help narrow down the options, but it doesn’t establish that a model is safe for clinical use. That requires testing in the specific clinic, using its tasks, rules, and constraints. The next steps are to broaden coverage, add tasks for patients and different regions, regularly check that benchmarks remain up to date, and account more accurately for the full cost of workflows.
Read next
Why Is It Difficult for an AI Agent to Act in the Real World?
WorldFoundry: How can you distinguish video generation from world
How AI Can Enhance an Individual’s Intelligence
FrontierPhysics: Let’s see whether AI agents can make discoveries
How AI Chooses the Right Model for Each Task
CaPtain: How to control a robot using a small AI model
WorkFactory: Hundreds of thousands of professional tasks helped train
How AI Agents Learn General Skills Across Different Tasks
How can you tell that an AI agent is doing the task incorrectly?
How an AI agent can master a new skill through practice using the Socratic method
How an AI agent improves interfaces on its own
How AI Predicts Brain Activity
AI reviews in simple way
Every day we read fresh AI papers and retell the essentials in plain human language — no hype, no fluff. If you want to see where AI agents are heading before everyone else, subscribe.
New reviews — every day
Follow on X