AI
“GPT-6-Astra solved 49 out of 50 unpublished research-level theoretical physics problems designed by world experts." (with supporting post text: 98% accuracy using Codex at under $3 per problem, Claude Opus 5.5 at 92%, Gemini 3.8 Flash at 68%, five agents evaluated, autonomous with tools, problems not requiring major breakthroughs, citing…”
Plain restatementA preprint benchmark study reports that the GPT-6-Astra model, run inside the Codex agent harness, produced correct final answers to 49 of 50 previously unpublished research-level theoretical physics problems contributed by subject-area physicists, with Opus 5.5 at 46/50 and Gemini 3.8 Flash at 34/50, at an average API cost below USD 3 per problem.
This one largely checks out. A real preprint, arXiv 2610.16733, by a group of academic physicists reports that OpenAI's GPT-6-Astra, run inside the Codex coding agent, got the right final answer on 49 of 50 previously unpublished research-level theoretical physics problems, with Claude Opus 5.5 at 92 percent and Gemini 3.8 Flash at 68 percent, at under three dollars per problem. Three things the post leaves out matter. The 98 percent figure is from the condition where the agent could search the web; with web search turned off Astra scored 46 of 50. Each agent was run only once per problem, so the gap between first and second place is small relative to normal variation. And the authors say their score only reached 98 percent after they found errors in their own reference answers and corrected them, which no outside party has yet rechecked. The paper itself is days old, not peer reviewed, and states plainly that none of the problems is a breakthrough open question.
GPT-6-Astra [drifted from the evidence:] solved 49 [drifted from the evidence:] out of 50 unpublished research-level theoretical physics problems [drifted from the evidence:] designed by [drifted from the evidence:] world experts." (with [drifted from the evidence:] supporting post text: 98% accuracy using Codex at under $3 per problem, Claude Opus 5.5 at [drifted from the evidence:] 92%, Gemini 3.8 Flash at [drifted from the evidence:] 68%, five agents evaluated, autonomous with tools, problems not requiring major breakthroughs, citing arXiv 2610.16733)
[added by the neutral restatement:] A preprint benchmark study reports that the GPT-6-Astra [added by the neutral restatement:] model, run inside the Codex agent harness, produced correct final answers to 49 of 50 [added by the neutral restatement:] previously unpublished research-level theoretical physics problems [added by the neutral restatement:] contributed by [added by the neutral restatement:] subject-area physicists, with Opus 5.5 at [added by the neutral restatement:] 46/50 and Gemini 3.8 Flash at [added by the neutral restatement:] 34/50, at an average API cost below USD 3 per problem.
Red-tinted words in the claim drifted from the evidence. Green-tinted words are what a neutral restatement needs.
The trace / claim to source
- The cited paper exists at the exact identifier given. arXiv:2610.16733, "TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving," is a real preprint.
- The headline number is the paper's own: 49 of 50 problems solved by GPT-6-Astra in Codex on the realistic track, reported as 98%.
- The secondary figures are correct for the same track: Opus 5.5 at 46/50 (92%) and Gemini 3.8 Flash at 34/50 (68%).
- Five agents were evaluated, as the post says.
- The cost figure is the paper's: under USD 3 per problem on average for Astra.
- The problems were unpublished at the time of testing and were contributed by physicists working in the relevant subfields.
- The difficulty calibration is the authors': a leading expert in the specific subfield would need roughly a day or less for a typical problem.
- The post's caveat that the problems did not require major new scientific breakthroughs reflects the paper's own statement.
- The model named is real and publicly available, announced by OpenAI on September 3, 2026.
- The post gives 98% with no mention that this is the realistic track, where the agent had live web search and literature access. On the isolated track, with web search disabled, Astra solved 46/50 rather than 49/50. The post also omits that each agent was run once per problem, so the scores are single-run pass rates and the 49-versus-46 ordering carries sampling noise.
- The post does not mention that the 98% figure follows the authors correcting their own reference solutions. The paper states that Astra was initially assessed as failing several additional problems and that, after the experts corrected their solutions, Astra's answers were found to be correct. A co-author describes the pre-correction score as in the late eighties. This is the authors' account of their own grading and has not been audited by an outside party.
- Grading was on converted final numerical answers, not on the correctness of the full derivation. A correct final number is the pass criterion in this benchmark.
- The post's closing line, that the study shows AI agents can now solve complex scientific problems and could speed up discoveries, stretches a 50-problem, single-run result on calculations the authors deliberately scoped to exclude breakthroughs into a general statement about scientific problem solving. The post does partly fence this with the no-breakthroughs sentence, which limits the overreach.
- The phrase "designed by world experts" slightly overstates the sourcing. The paper describes problems supplied by expert contributors, in some cases taken from research notes they had worked out but not published, with the moderators then fine-tuning the problems for the benchmark.
- Whether the corrected reference solutions are themselves right. No independent party has rechecked either the corrected answer keys or the model outputs, so the 98% rests on the question-setters' own revised grading.
- Whether the result replicates. The evaluation is a single run per problem per agent, and no outside re-run has been published.
- Whether the preprint survives review. It is arXiv v1, posted days ago, and unrefereed.
- How much of the cross-model ordering reflects the underlying models versus their differing agent harnesses, since each model was run inside its own vendor's scaffold.
- Whether any vendor provided compute credits or API access for the runs. No such statement was found in the acknowledgment text retrieved, and absence here is not proof either way.
The cited paper is real, the arXiv identifier in the post is correct, and the headline figure is the paper's own. The paper reports that on the realistic track Astra has the highest pass rate, solving 49/50 problems, closely followed by Opus, which solved 46/50 problems, while Gemini solved 34/50 and the open-weight models DeepSeek and GLM solved 27/50 and 21/50 respectively. Those convert to 98%, 92%, 68%, 54% and 42%. The exact systems match the post. The authors evaluated five agents: GPT-6-Astra with Codex (Astra); Opus 5.5 with Claude Code CLI (Opus); Gemini 3.8 Flash with Antigravity (Gemini); DeepSeek-V4.1-Flash with Codex (DeepSeek); and GLM 5.3 with Claude Code CLI (GLM). Reasoning effort was set to max for all models except Gemini 3.8 Flash, where "high" is the highest available setting. The cost and conditions also match. The abstract states that when given access to tools available in a realistic setting, including the ability to search the literature and write scripts for numerical and symbolic work, the best-performing agent, GPT-6-Astra harnessed in Codex, obtains final answers with near-perfect accuracy of 98% at an average cost under USD 3 per problem, and that agents show only small changes in accuracy when denied web search and access to background research papers, suggesting significant memorization of the literature. Two conditions were tested. In the realistic track the agent had access to both web search and local tools; in the isolated track the agent could use local tools but web search was disabled except for calls to the model provider itself. On the isolated track Astra and Opus had the highest scores, solving 46/50 problems each, followed by Gemini at 31/50, GLM at 26/50 and DeepSeek at 25/50. The paper is explicit that the headline score rests on reference solutions the authors themselves revised. It states that the authors initially assessed that Astra had made errors in several additional problems, and that in all these cases, after the experts corrected their solutions, Astra's solution was found to be correct. A co-author's public summary describes the pre-correction picture: "Our initial evaluation suggested that GPT-6-Astra's score was in the late eighties followed closely by Opus 5.5." Grading was on final numerical answers, not full derivations. The paper notes that agent solutions were graded on numerical answers, and that after the final runs an automated analysis revealed two prompts where conventions were not fully specified, with the grading key allowing for the convention choices observed in the runs. The paper also notes that the single failure on the realistic track is an inverse problem, which involves information asymmetry between the agent and the creator of the question. On difficulty framing, the authors state that a leading expert in the relevant subfield would need about a day or less to solve a typical problem without AI help, and that specialists from neighboring fields would take much longer, and the paper is explicit that none of its problems is a breakthrough open problem and that solving them requires familiarity with current research rather than a new theory. Several agents do not saturate the evaluation: DeepSeek-V4.1-Flash scored 54% and 50% on the realistic and isolated tracks. The model named in the claim exists and is shipping. OpenAI announced the rollout of GPT-6 Astra on September 3, 2026, and the developer documentation lists gpt-6-astra as a selectable API model.
Complete reasoning
The reply is formatted for pasting into the thread where the claim is circulating.
Ask this case
Answers come only from the case file above; nothing is added.
Did GPT-6-Astra really solve 49 out of 50 physics problems?
Yes, that figure comes from the actual preprint, arXiv 2610.16733, which reports GPT-6-Astra running in the Codex agent got 49 of 50 correct on what the authors call the realistic track, where the agent had web search and tool access.
Does the 98% score hold up if the AI can't search the internet?
No. With web search disabled, on the isolated track, GPT-6-Astra's score dropped to 46 out of 50, the same as Opus 5.5 on that track. The post only cites the realistic-track number.
Were these problems graded on full solutions or just final answers?
Grading was based on final numerical answers, not the correctness of the full derivation. The paper also notes two prompts had ambiguous conventions that the grading key had to account for after the fact.
Is the 98% figure fully independent and verified?
Not entirely. The paper states that Astra was initially judged to have failed several more problems, and only after the authors corrected their own reference solutions were those answers counted as correct. A co-author described the pre-correction score as in the high 80s. This self-correction has not been independently audited.
Does this mean AI can now solve real scientific breakthroughs?
No, the investigation found the paper explicitly states none of the 50 problems is a breakthrough open question, and solving them requires familiarity with current research rather than new theoretical insight.