AI
“Microsoft's Diagnostic Orchestrator AI system scored 83% accuracy on published complex medical cases, compared with 36% for doctors using references and 22% without them.”
Plain restatementOn a set of published complex medical cases, Microsoft's Diagnostic Orchestrator (MAI-DxO) reached a diagnostic accuracy of about 83%, while generalist physicians reached about 36% when permitted reference materials and about 22% when not.
This one checks out on the numbers. A peer-reviewed paper in the journal Science, published on 22 October 2026 by Microsoft researchers, reports that their Diagnostic Orchestrator running on GPT-5 reached 83.2% accuracy on 304 difficult published case records, while generalist doctors reached 36.1% with reference materials and 22.3% without them. The missing context matters, though. The doctors worked alone on text-only transcripts with no scans, no physical examination and no colleagues to consult, and the paper itself says the results do not show the system beats specialists, medical teams or real clinical practice. The version that scored 83% also ordered about $5,400 in tests per case, slightly more than the doctors it was compared with, so the cheaper-than-doctors point applies to other settings of the system rather than this one. The cases were teaching cases picked to be hard, with no healthy patients included, so nobody yet knows how the system behaves on ordinary complaints or how often it raises false alarms. Microsoft has said hospital testing in Boston is planned but has not happened, and no outside group has independently repeated the test.
[drifted from the evidence:] Microsoft's Diagnostic Orchestrator AI system scored 83% accuracy on published complex medical cases, [drifted from the evidence:] compared with 36% [drifted from the evidence:] for doctors using references and 22% [drifted from the evidence:] without them.
On [added by the neutral restatement:] a set of published complex medical cases, [added by the neutral restatement:] Microsoft's Diagnostic Orchestrator (MAI-DxO) reached a diagnostic accuracy of about 83%, while generalist physicians reached about 36% [added by the neutral restatement:] when permitted reference materials and [added by the neutral restatement:] about 22% [added by the neutral restatement:] when not.
Red-tinted words in the claim drifted from the evidence. Green-tinted words are what a neutral restatement needs.
The trace / claim to source
- The 83% figure is real and traces to a refereed paper, not a press release. *Science* reports 83.2% accuracy on NEJM clinicopathological conference cases, against generalist physicians at 36.1%
- The two physician figures are correctly reported and correctly labelled. The manuscript gives 22.3% for the closed-book arm and 36.1% for the open-book arm, matching the post's "using references" and "without them" framing
- The post's attribution to GPT-5 is correct for this figure. The 83.2% result is DxO with GPT-5 in the configuration without budget constraints
- The post's secondary claim about emergency-room cases is supported. Generalist physicians averaged 46.8% on ED cases versus 36.1% on CPC cases, against 80.3% for the system
- The post's own caveat is accurate and matches the paper. *Science* itself states the findings were obtained in a controlled benchmark setting, and Microsoft has said clinical validation at Beth Israel Deaconess Medical Center in Boston is planned
- The claim says "83% accuracy on published complex medical cases" without the condition the paper attaches to exactly that comparison. The paper specifies controlled, text-only conditions against individual generalist physicians, and states the comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice. A reader of the sentence alone would not know the physicians worked alone on text transcripts with no imaging, no examination and no colleagues. The post's slides do carry a real-world caveat, which softens this, but the sentence as circulated does not.
- The AI percentage and the physician percentages are not computed over the same cases. AI agents were scored on all cases while physician performance covers only the 56 held-out test-set cases. The direction and size of the headline gap are not in question, but the three numbers are not the strict like-for-like triple the phrasing implies.
- The 83% figure carries a price tag the claim does not mention. The configuration reaching 83.2% did so at an average cost of $5431 per case, against $4,725 per case for the open-book physicians. The post's caption says the system "estimated it could reach diagnoses at a lower cost," which is true of the budget-constrained configurations but not of the one that produced the 83% headline.
- Marketing as evidence, partially: the figures are vendor-produced. The benchmark, the information gatekeeper, the grading judge and the system are all from the same team, and no independent party has re-run the evaluation. Publication in *Science* raises this well above a vendor blog post, but peer review of a manuscript is not independent replication of a result.
- Capability extrapolation, in the surrounding post rather than the claim sentence: the headline reproduced on the slides, that the tool "can diagnose diseases up to 4x better than doctors," rests on the 22% arm in which doctors were deliberately deprived of reference material. Microsoft has described its clinician cohort as working without access to colleagues, textbooks or other resources. The 4x ratio therefore describes a comparison against resource-deprived generalists, not against doctors as they ordinarily practise. The claim sentence under review is more careful than the headline above it, because it states both arms.
- Whether the result holds on routine presentations is unresolved by design. The authors state they do not know whether performance gains on hard cases generalize to common, everyday clinical conditions, and that they could not measure false positive rates because the case set contains no healthy or benign presentations.
- How much of the gain is the orchestrator versus the underlying model is contested in the coverage. TechCrunch reports the study's own data suggests most of the improvement traces to the more powerful underlying models, which is consistent with GPT-5 alone scoring 78.9% at baseline.
- Residual contamination risk cannot be fully excluded for published NEJM cases, though the held-out split is a reasonable check and the authors report it held.
- No independent evaluator has reproduced the 83.2% figure, and none existed to check as of 2026-10-24.
- Real-world clinical performance is untested. The prospective hospital study has been announced, not completed.
The paper exists, is real, and is published in a refereed venue. The *Science* abstract states that the system achieved 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians at 36.1 and 46.8% respectively, and adds that the findings were obtained in a controlled benchmark setting. The Results section specifies that DxO with GPT-5 established a Pareto frontier that outperformed participating physicians at every cost or accuracy level, and that the configuration without budget constraints achieved 83.2% accuracy at an average cost of $5431. The author manuscript gives the physician arms separately. It reports that physicians' average diagnostic accuracy was 22.3% (closed-book) and 36.1% (open-book), and that on NEJM-CPC a cohort of US and UK physicians with a median of 12 years of experience achieved 36.1% accuracy at an average cost of $4,725 per case. In the closed-book arm, physicians were explicitly instructed not to use external resources including search engines, language models, or other online sources of medical information, in order to prevent them from finding the NEJM cases online. The study's own scoping language is explicit. The paper states that DxO operated at higher regions of the estimated cost-accuracy frontier when compared with participating individual generalist physicians under controlled, text-only conditions, and that these comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice. On the comparison sets: AI agents were evaluated on all cases, while physician performance is shown only for the 56 held-out test-set cases. In the earlier version of the work, the most recent 56 cases from 2024 to 2025 were held out as a hidden test set and remained unseen during development. Independent coverage adds one material point the post does not: TechCrunch reports that the study's own data suggests much of the improvement comes from more powerful underlying models, with Microsoft's orchestration adding gains on top, and that Microsoft tested DxO with models from several companies but got its best results with GPT-5 and GPT-5.5. The manuscript supports that reading: GPT-5 reached the highest baseline accuracy of 78.9% on NEJM-CPC at a cost of $8,409, while MAI-DxO achieved 83.2% at $5,430, or 84.5% at $8,945.
Complete reasoning
The reply is formatted for pasting into the thread where the claim is circulating.
Ask this case
Answers come only from the case file above; nothing is added.
Did Microsoft's AI system really score 83% versus 36% and 22% for doctors?
Yes, those numbers come from a peer-reviewed paper in Science. The system (MAI-DxO with GPT-5) scored 83.2% on hard published cases, while physicians scored 36.1% with reference materials and 22.3% without them.
Were the AI and the doctors tested under the same conditions?
Not exactly. The doctors worked alone on text-only case transcripts with no scans, no physical exam and no colleagues to consult, and their results are reported only for 56 held-out test cases, while the AI was scored on the full case set.
Does this mean the AI is better than real doctors in practice?
No. The paper itself states these comparisons do not show the system beats specialists, medical teams, or real-world clinical practice. The results come from a controlled benchmark setting, not a hospital.
Was the AI also cheaper than the doctors?
Not in this particular comparison. The configuration that scored 83% cost about $5,431 per case, slightly more than the $4,725 per case for the doctors who could use references. Cheaper results apply to other, budget-constrained versions of the system.
How much of the improvement comes from the orchestrator itself versus the underlying AI model?
The case file notes this is disputed. Reporting cited in the file says much of the gain may come from the more powerful underlying model (GPT-5) rather than Microsoft's orchestration layer, though the orchestrator did add further gains on top.