AI
“Microsoft's AI diagnosed patients roughly twice as accurately as doctors, according to a study published in Science.”
Plain restatementA study published in the journal Science reports that a Microsoft AI diagnostic system reached correct diagnoses at approximately twice the rate of physicians.
The study is real. Science published it on 22 October 2026, written by Microsoft AI researchers. Its system reached a correct diagnosis in 83.2% of 304 complex published case records, against 36.1% for generalist doctors who could use textbooks and the internet, and 80.3% against 46.8% on emergency department records. So "roughly twice" is a fair description of the reported gap. The misleading part is the word "patients." No patients were involved. Both the AI and the doctors worked through written historical case files in a simulator, with no physical examination and no imaging, and the doctors in the comparison were general physicians working alone, not specialists or hospital teams. The authors say in the paper itself that these results do not establish that the system is better than specialists, medical teams, or real-world practice. A hospital study with real patients is planned but has not happened yet, and no outside group has yet repeated the test.
[drifted from the evidence:] Microsoft's AI diagnosed patients roughly twice as accurately as doctors, according to a study published in Science.
A study published in [added by the neutral restatement:] the journal Science [added by the neutral restatement:] reports that a Microsoft AI diagnostic system reached correct diagnoses at approximately twice the rate of physicians.
Red-tinted words in the claim drifted from the evidence. Green-tinted words are what a neutral restatement needs.
The trace / claim to source
- A study was published in Science on 22 October 2026 reporting these results, and it is a Microsoft AI paper. The journal, date and institution in the claim are all correct.
- The ratio is in the right range and is, if anything, understated for the hardest cases. On the complex NEJM cases the reported figures are 83.2% for the system against 36.1% for generalist physicians using references, which is 2.3 times. On emergency department cases they are 80.3% against 46.8%, which is 1.7 times.
- Microsoft's own summary states the system is 1.7 times more accurate than physicians on common US emergency department presentations from MIMIC-IV-ED, and more than twice as accurate as physicians using textbooks and the internet on complex NEJM cases, and nearly four times as accurate when those physicians work without such resources.
- Every specific number in the accompanying post checks out against the paper: 304 complex cases, 83% for the AI, 36% for generalists with reference materials, roughly 52% for the best individual physician (51.8%), and 80% against 47% on emergency department cases (80.3% against 46.8%).
- The post's description of a five-role virtual panel that proposes diagnoses, orders tests, challenges conclusions, controls cost and checks quality matches the paper's architecture.
- The two caveats stated in the post, that the comparison was against generalists rather than specialists and that the test used written case descriptions rather than real patients, are both correct and are stated by the authors themselves.
- The claim says the AI "diagnosed patients." It did not. It worked through historical, text-only case records inside a simulator, with a software Gatekeeper releasing information on request and in some instances generating plausible test results that were not in the original case. Neither the AI nor the physicians examined a patient or viewed an image. The paper's authors say directly that the comparisons do not establish superiority in real-world clinical practice. A reader of the claim alone would reasonably picture clinical encounters, which is not what happened.
- The claim says "doctors," unqualified. The comparison group was individual generalist physicians working alone. No specialists, no multidisciplinary teams, no colleague consultation and no electronic health record. On the exact case type where the 2.3x gap appears, complex published case conferences, specialist input is the normal clinical pathway, so the omitted word "generalist" is the word doing most of the work in the gap.
- "roughly twice" collapses two different comparisons with different ratios and silently drops the condition that produced them. Against physicians allowed references the gap is 2.3x on complex cases and 1.7x on emergency cases. Against physicians denied references it is 3.7x. The number you quote depends entirely on which resource condition and which case set you pick, and the claim names none of them.
- On the complex cases the two numbers do not come from identical case sets. The AI figure is drawn across the full case collection while the physician figure comes from a 56-case held-out subset, and the authors themselves noted in the earlier version of this work that AI agents score better on that subset than on the full set. The comparison is presented as head to head and is not strictly like for like.
- The claim presents accuracy alone. The system's 83.2% run carried an average estimated workup cost of $5,431 per case against $4,725 for open-book physicians, so at its top accuracy setting the system was not the cheaper option, even though cheaper configurations with slightly lower accuracy are also reported.
- Misattribution, partial: "Microsoft's AI" is fair for the orchestration layer, which is Microsoft's, but the headline result was produced with OpenAI's GPT-5 underneath. The system is model-agnostic scaffolding, not a Microsoft-built model.
- Whether the gap holds outside the benchmark. The case sets exclude healthy and benign presentations, so the system's false positive rate on well patients is not measured by this study. A real-world study at Beth Israel Deaconess Medical Center is reported as planned, and its results do not exist yet.
- Whether any party other than the authors can reproduce the physician comparison. The benchmark code was released on 22 October 2026 under an MIT license, but I found no independent run of it as of 28 October 2026.
- How much benchmark contamination remains. The authors mitigate it by holding out recent post-cutoff cases, but published case conferences are widely discussed online and no external audit of contamination was located.
- Whether the physicians' 36.1% reflects their ceiling. They worked alone under time and interface constraints inside a vendor-built simulator, and the paper does not establish what the same physicians would score with their ordinary tools and colleagues.
The study is real and is where the claim says it is. Science published "Sequential diagnosis with language models" by Nori et al. on 22 Oct 2026, Vol 394, Issue 6822, pp. 423-428, DOI 10.1126/science.aeh7930. The headline numbers are accurately reported. The abstract states that the system achieved 83.2% accuracy on New England Journal of Medicine clinicopathological conference cases and 80.3% on MIMIC-IV-ED emergency department admissions, outperforming generalist physicians at 36.1% and 46.8% respectively. That is a ratio of 2.3x on the complex cases and 1.7x on the emergency department cases. The physician comparison conditions are documented in the paper. The authors recruited active primary-care and hospital and acute-care physicians, ran the CPC physician study on 56 cases held out from engineering efforts, collected 764 closed-book and 662 open-book diagnoses, and recorded average accuracies of 22.3% closed book and 36.1% open book. The AI's headline figure comes from the full set: the benchmark transforms 304 NEJM clinicopathological conference cases into stepwise encounters, and the paper's own figure shows physicians for the test set only. In the earlier preprint version the authors noted that AI agents are evaluated on all 304 cases while physician performance is shown only for the held-out 56, and that AI agents tend to perform better on this subset than on the 304. The underlying model is not Microsoft's. The manuscript states that when paired with OpenAI's GPT-5, the orchestrator achieved 83.2% on NEJM-CPC and 80.3% on MIMIC-IV-ED, compared with 36.1% and 46.8% by generalist physicians. Run without budget constraints it reached 83.2% at an average cost of $5,431 per case, while open-book physicians averaged 36.1% at $4,725 per case; the single best physician reached 51.8%, and on the emergency-department cases physicians averaged 46.8% with the best individual at 62.5%. Most importantly, the paper itself bounds what the comparison establishes. The Science text states that the result holds "when compared with that of participating individual generalist physicians under controlled, text-only conditions. These comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world" clinical practice. Reporting on the paper quotes the same qualifier: the researchers state the result does not mean the AI is better than doctors in real-world practice, writing that these comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice. No patients were involved. The paper's limitations section notes that the benchmark currently includes only pathological cases drawn from CPC cases and ED encounters, that CPC cases may overrepresent diagnostically challenging presentations, and that both exclude healthy or benign encounters. Coverage notes the cases were presented as text and that neither the AI nor the doctors had access to physical examinations or medical imaging. The post's description of the system's architecture is accurate. A single language model role-plays five distinct medical personas: Dr. Hypothesis maintains a probability-ranked differential, Dr. Test-Chooser selects up to three discriminating tests per round, Dr. Challenger acts as devil's advocate against anchoring bias, Dr. Stewardship enforces cost-conscious care, and Dr. Checklist performs silent quality control.
Complete reasoning
The reply is formatted for pasting into the thread where the claim is circulating.
Ask this case
Answers come only from the case file above; nothing is added.
Did a Microsoft AI really score about twice as accurately as doctors in a Science study?
A study in Science did report this kind of gap: 83.2% accuracy for the AI system versus 36.1% for generalist physicians on complex cases, and 80.3% versus 46.8% on emergency department cases. So the numbers are real, but the comparison behind them needs context.
Did the AI actually diagnose real patients?
No. Both the AI and the physicians worked through written historical case records in a text-only simulator. There was no physical examination, no imaging, and no real patient involved.
Were the doctors in the study specialists or ordinary physicians?
They were generalist physicians working alone, allowed to use textbooks and the internet but without colleagues, specialists, or electronic health records. The study did not compare the AI against specialists or medical teams.
Do the study's authors say the AI is better than doctors in real clinical practice?
No. The paper states directly that these comparisons do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice.
Has this result been confirmed with real patients or by outside researchers?
The case file states a hospital study with real patients is planned but has not happened yet, and no outside group has repeated the test so far.