What the study measured
Researchers tested MAI-DxO on two datasets: 304 clinicopathological cases from the New England Journal of Medicine and 325 emergency-department-to-hospital admissions from MIMIC-IV-ED.
On the first set, DxO’s diagnostic accuracy was 83.2%, compared with 36.1% for general practitioners who had access to reference materials. On the second, the system scored 80.3%, against 46.8% for doctors.
Microsoft also compared the estimated cost of tests needed to reach a diagnosis. The company said DxO generally achieved its results at half or even a third of the doctors’ testing costs.
The system worked through each case in stages: it received an initial complaint, proposed a preliminary diagnosis, chose follow-up questions and decided which tests to order while accounting for their cost. The study’s authors say earlier research did not test diagnosis as a sequential process that weighs both accuracy and testing cost; instead, models were given the full medical history in a single prompt.
The patient was still simulated
That staged design addresses one important limitation of earlier tests, but it does not reproduce a clinical encounter. DxO got its requested details from an intermediary agent that relayed information from the medical record. The patient, in effect, was another AI. And doctors had prepared the detailed records the model analyzed.
That leaves a central part of diagnosis untested: helping a patient explain what is happening and drawing out details they may not know are relevant. The study measured how well the model reasoned over information made available to it, not how well it could elicit that information from a person.
Microsoft presented the findings as a win. Mustafa Suleyman, the company’s head of AI, told The Verge that the era of improving healthcare with AI had “finally arrived.” He predicted impressive deployments of intelligent systems next year that would substantially change how care is delivered.
I think the gap between the benchmark and the clinic is the more consequential result here. The scores suggest the system may reason effectively over carefully prepared cases; they do not establish that it can gather the right evidence in a real consultation. Until that part is tested, lower estimated testing costs are a claim about a simulated workflow, not a demonstrated saving in patient care.
Daily AI news
Every day we pick what actually matters in AI and explain it plainly — no hype, no filler. Subscribe if you want to follow where the industry is going.
Only what matters — every day
Follow on X