AI
“Microsoft published initial results showing its MAI-DxO AI diagnostic system beats physicians on both complex medical cases and everyday ER visits while ordering fewer, cheaper tests.”
Plain restatementMicrosoft has published evaluation results in which its MAI Diagnostic Orchestrator (MAI-DxO / DxO) scored higher than participating physicians on two case sets, one of complex NEJM clinicopathological conference cases and one of emergency department cases, and did so at lower testing cost and with fewer tests ordered.
Microsoft really did publish this research, in the peer-reviewed journal Science on 22 October 2026, and the accuracy numbers in the post are correct: its MAI-DxO system scored 83.2 percent on 304 complex New England Journal of Medicine teaching cases against 36.1 percent for generalist doctors allowed to use references, and 80.3 percent against 46.8 percent on 325 emergency department admission records. The part about doing this "while ordering fewer, cheaper tests" does not match the paper. The runs that produced those two headline scores actually cost more per case than the doctors they were compared with, about 5,431 dollars versus 4,725 on the complex cases and about 4,979 versus 3,400 on the emergency cases. Cheaper versions of the system do exist and still beat the doctors, but they score lower, around 81 percent and 73 percent, so the post credits one result with both the best accuracy and the cost saving. Other context is missing: this was a text-only simulation using historical written case records, with no real patients, no physical exams and no healthy patients to test false alarms, and the human comparison was individual general practitioners and hospital generalists rather than specialists or medical teams. The study was designed, run and reported by Microsoft, and no independent group has reproduced it. A real-patient study at Beth Israel Deaconess in Boston has been announced but has not reported any results.
Microsoft published [drifted from the evidence:] initial results [drifted from the evidence:] showing its [drifted from the evidence:] MAI-DxO AI diagnostic [drifted from the evidence:] system beats physicians on [drifted from the evidence:] both complex [drifted from the evidence:] medical cases and [drifted from the evidence:] everyday ER visits while ordering fewer, [drifted from the evidence:] cheaper tests.
Microsoft [added by the neutral restatement:] has published [added by the neutral restatement:] evaluation results [added by the neutral restatement:] in which its [added by the neutral restatement:] MAI Diagnostic [added by the neutral restatement:] Orchestrator (MAI-DxO / DxO) scored higher than participating physicians on [added by the neutral restatement:] two case sets, one of complex [added by the neutral restatement:] NEJM clinicopathological conference cases and [added by the neutral restatement:] one of emergency department cases, and did so at lower testing cost and with fewer tests [added by the neutral restatement:] ordered.
Red-tinted words in the claim drifted from the evidence. Green-tinted words are what a neutral restatement needs.
The trace / claim to source
- Microsoft did publish these results. The paper appeared in *Science* on 22 October 2026 and is peer reviewed, building on a June 2025 arXiv preprint of an earlier o3-based version.
- On the 304 NEJM clinicopathological conference cases, the paper reports 83.2% accuracy for DxO running on GPT-5, against 36.1% for the generalist physicians who had reference tools and 22.3% for those who did not. "More than doubled physicians' accuracy with reference tools" is a fair reading of 83.2 versus 36.1.
- On the emergency department case set, the paper reports 80.3% for DxO against 46.8% for physicians, matching the post's "80% vs. 47%."
- The post's "one setup reached 81% accuracy at less than half the cost" tracks a real row in the paper: 80.9% at $2,241 per case, which is below the open-book physician average of $4,725 and far below plain GPT-5 at about $8,409.
- The paper does report that DxO achieved higher accuracy than physicians and off-the-shelf models at any cost configuration tested, and budget-limited configurations did beat physician accuracy at lower estimated cost in both case sets.
- The system does operate as described, a single model prompted into multiple panel roles that propose diagnoses, select tests and challenge each other.
- Microsoft has announced a prospective evaluation with Beth Israel Deaconess Medical Center in Boston, with performance to be measured against clinicians' independent assessments.
- The claim attaches "fewer, cheaper tests" to the same result that beat physicians, but the configurations that produced the quoted 83.2% and 80.3% figures cost more per case than the physicians they were compared with, $5,431 against $4,725 on the complex cases and $4,979 against $3,400 on the emergency cases. The cheaper runs are different, lower-scoring configurations: 80.9% at $2,241 and 79.6% at $1,303 on the complex cases, 73.2% at $2,634 and 69.5% at $2,037 on the emergency cases. Those cheaper runs still beat the physicians on accuracy, so a cheaper-and-more-accurate result does exist, but it is not the result the headline numbers come from. Presented as one combined win, the claim credits a single run with both the top accuracy and the cost saving.
- "ordering fewer tests" is the vendor blog's phrasing and is not supported in the material retrieved for the configurations that produced the headline scores. The paper gives physician test counts of 7.1 and 12.1 per complex case; a corresponding DxO count was not found in what I retrieved, and the higher per-case cost of the unconstrained runs points the other way.
- "everyday ER visits" and "typical ER cases" describe 325 emergency department-to-admission encounters drawn from the MIMIC-IV-ED database, meaning patients sick enough to be admitted, assembled from historical records. The authors also note the case sets include no healthy patients or benign presentations, so the evaluation could not measure how often the system raises a false alarm, which is central to what "everyday" visits involve.
- "beats physicians" compresses a text-only simulation on historical written case records into a general statement about clinical performance. There were no real patients, no physical examination, no live imaging and no treatment decisions. Microsoft itself frames the hospital study as the real test, and the post's own body text acknowledges this, but the headline claim does not.
- The comparator was primary care physicians and in-hospital generalists working individually, with cohorts of 21, 19 and 16 people, no specialists, no teams, no ability to consult colleagues. The complex NEJM cases are the type normally worked up by specialist teams. "Physicians" in the claim reads as physicians generally, which the study design cannot support. Clinicians quoted on the 2025 preprint made this same objection about the closed-book version, and the 2026 paper's open-book arm partially answers it: the gap narrowed from roughly 4x to roughly 2.3x once doctors could use books and the internet.
- The evaluation, the benchmark, the system and the physician baselines all come from one interested party. Peer review at *Science* means the work passed referees; it does not mean an independent group has reproduced the result. No independent replication was found.
- Omitted qualifier (on the post's secondary GPT-5.5 point): "the gains holding even with newer models like GPT-5.5" holds on the complex cases, where the orchestrator lifted GPT-5.5 from about 72% to nearly 87%, but not on the emergency cases, where GPT-5.5 scored about 85% with and without the orchestrator at similar cost. More broadly, the paper's own data show most of the performance comes from the underlying model rather than Microsoft's orchestration layer: plain GPT-5 already scored 78.9% on the complex cases.
- Whether DxO ordered fewer tests than physicians, in any configuration, is unresolved. Physician test counts are published; a matching DxO figure was not located, and the cost figures for the headline configurations point against it.
- Whether the accuracy advantage survives contact with real patients is untested. That is the stated purpose of the planned Beth Israel Deaconess study, which has no published results and, per one report, no firm start date beyond "the next couple of months."
- Whether the NEJM CPC cases, which are published and widely discussed online, are free of contamination in the training data of the models evaluated. The authors took steps to gate information behind a gatekeeper model, but I found no published contamination analysis for the 2026 results.
- How the modeled cost estimates relate to real spending. The prices are chargemaster-derived estimates and exclude the downstream cost of a wrong diagnosis, regional variation, insurer rates, test invasiveness and wait times.
- The exact instructions given to the 16 physicians in the emergency department arm, and whether all had emergency medicine backgrounds, were not established from the extracts retrieved.
The paper exists, is peer reviewed, and was published in *Science* on 22 October 2026. Its abstract states that the system achieved 83.2% accuracy on NEJM clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, against 36.1% and 46.8% respectively for generalist physicians. The benchmark, SDBench, converts 304 NEJM CPC cases and 325 emergency department-to-admission encounters from MIMIC-IV-ED into stepwise encounters in which the agent or physician requests information from a gatekeeper model. Scoring covers both diagnostic accuracy and the cumulative estimated cost of tests ordered, priced from CPT codes on a major health system's chargemaster, plus a fixed $300 physician-visit cost. On cost, the paper reports a configuration frontier rather than a single number: - NEJM cases: DxO with GPT-5 unconstrained reached 83.2% at $5,431 per case. Budget-constrained variants reached 80.9% at $2,241 and 79.6% at $1,303. A maximum-accuracy variant reached 84.5% at $8,945. Physicians averaged $4,725 (open book, 36.1%) and $3,049 (closed book, 22.3%). - ED cases: DxO unconstrained reached 80.3% at $4,979 per case. Budget-constrained variants reached 73.2% at $2,634 and 69.5% at $2,037. Physicians averaged $3,400 at 46.8%. So in both settings, the configurations that produced the headline accuracy figures cost MORE per case than the physicians they were compared with. The paper separately reports Pareto dominance, meaning that at any given cost level DxO scored higher than both off-the-shelf models and physicians, which is how the cheaper-and-still-better configurations arise. On test counts, the paper reports physicians requesting 7.1 tests (closed book) and 12.1 tests (open book) per NEJM case. A per-case test count for DxO did not appear in the extracts retrieved; one secondary write-up reports the paper saying DxO asked about 25.5 questions before ordering its first test, versus 5.6 for physicians. On where the gain comes from, GPT-5 alone with a basic prompt scored 78.9% on the NEJM cases at roughly $8,409; the orchestrator added about four accuracy points and removed roughly $3,000 of cost. Averaged across all models tested, orchestration added 7.9 points on NEJM cases and 4.4 points on ED cases. With GPT-5.5, the orchestrator lifted complex-case accuracy from about 72% to nearly 87%, but on ED cases GPT-5.5 scored about 85% with and without the orchestrator at similar cost. Microsoft's own blog post frames the results as achieving the right diagnosis "at half or even one third of the cost" with "fewer tests and procedures." Microsoft also announced a prospective study with Beth Israel Deaconess Medical Center in Boston, in which patients will interact with the system and its performance will be compared against physicians' independent assessments.
Complete reasoning
The reply is formatted for pasting into the thread where the claim is circulating.
Ask this case
Answers come only from the case file above; nothing is added.
Did Microsoft really publish these results?
Yes. The research appeared in a peer-reviewed paper in Science on 22 October 2026, building on an earlier 2025 preprint.
Did the AI system actually beat physicians on accuracy?
Yes. On 304 complex NEJM cases it scored 83.2 percent versus 36.1 percent for physicians using references, and on 325 emergency department cases it scored 80.3 percent versus 46.8 percent.
Did the system also order fewer, cheaper tests while getting those scores?
No. The runs that produced the 83.2 percent and 80.3 percent scores actually cost more per case than the physicians they were compared with, about 5,431 dollars versus 4,725 and about 4,979 versus 3,400. Cheaper versions of the system exist and still beat physicians, but they score lower, around 81 percent and 73 percent, so the claim wrongly credits one result with both the top accuracy and the cost saving.
Does this mean the system would work the same way on real patients in an ER?
The investigation did not establish this. The case sets were text-only simulations built from historical written records, with no real patients, no physical exams, and no healthy patients included, so there is no way to measure false alarms, which matters for everyday ER visits.
Has anyone outside Microsoft confirmed these results?
No independent group has reproduced the study. Microsoft designed, ran, and reported it, and a planned real-patient study with Beth Israel Deaconess Medical Center has been announced but has not yet reported results.