TheoPhysEval: AI Scores 98% on Unpublished Theoretical Physics Problems, Corrects Experts

Full analysis

A new benchmark called TheoPhysEval put 50 unpublished theoretical physics problems in front of frontier models. GPT-6-Astra from OpenAI scored 98%. Claude Opus 5.5 from Anthropic landed just behind. The result everyone is quoting: when the models disputed some of the expert-supplied answers and held their ground, the physicists went back, checked, and found calculation errors in their own work. The AI's score went up as the humans fixed their mistakes. Jack Clark is presenting this as proof that the models have crossed into real scientific creativity, because unpublished problems can't have been in the training data.

What's actually being decided here is not whether to buy anything. It's what to believe about where frontier reasoning now sits, and whether that belief should change how you build, price, or trust AI in technical work. This is easy to undo as a belief, hard to undo as a habit: if your team starts deferring to the model on technical ground truth, that habit sticks. No deadline sets this one. It's a read on capability, and the only clock is the next benchmark.

The Skeptic. Fifty problems is a vignette, not a benchmark. "Unpublished" is not "novel." Theoretical physics runs on a small set of standard techniques and problem shapes that are drowning in the training data. A problem nobody printed can still be a textbook method pointed at a fresh setup. And the expert-error story, the part doing all the work, proves the model is a very good symbolic calculator. We knew that. Catching an algebra slip buried in a derivation is not inventing physics. Jack Clark calling this "genuine scientific creativity" is a large claim resting on 49 right answers out of 50. Ask how many of those problems needed a new mathematical structure versus a clever use of old ones. Nobody has answered that.

The Safety Lens. The dynamic to flag is the one everyone finds charming: the model insists, the expert defers, the expert turns out wrong. That trains people. Do it enough in physics, where being wrong costs a corrected footnote, and the deference generalizes. The same loop in drug synthesis, pathogen design, or exploit writing is where it stops being charming. A model that overrides credentialed human judgment with sustained technical confidence is a capability, and it does not come with a domain filter attached. We have no tooling to audit a hard-science reasoning chain at the trust level this result will hand the model. The win in a harmless domain is exactly what makes the risk in the dangerous ones invisible.

The Researcher. The score is not the finding. The benchmark had to be revised because the model caught the graders' mistakes. That means the ground truth moved, which is a genuinely strange place to be. Building evals for frontier-capability domains now needs the model under test to adversarially review the answer key before you publish, or your "errors" are the model being right. That is uncomfortable and probably unavoidable. The 50-problem count is too small to claim anything statistical. But the qualitative result holds: Suvrat Raju says no individual he knows could solve a meaningful fraction of these without AI help. Physics PhDs are a floor now, not a ceiling. The risk is reading a handful of corrected calculations as a paradigm shift because "AI corrects experts" is a story that writes itself.

The Builder. If the reasoning engine is this good, the right tool for hard science is long-context reasoning wired into a symbolic math environment, Mathematica or SymPy, so the model can actually compute instead of narrate. You can start building first-pass research agents, not just copilots. The thing that breaks at 90 days is the failure you can't see: a derivation that is wrong but plausible, polished enough to pass a casual expert read. The model is now good enough that its mistakes look like results. That moves the whole quality-assurance burden onto you, and your reviewers have to be at least as good as the model to catch it. On the 2% it misses, you will not get an obvious wrong answer. You will get a confident one.

The Compute Pragmatist. A 98% on graduate theoretical physics requires sustained multi-step symbolic reasoning over long context, extended thinking budgets, orders of magnitude more compute than a normal call. That prices hard-science research out of the commodity inference market and parks it at the labs with the scale to run it. OpenAI and Anthropic sit on that scale. Llama, Mistral, Qwen do not, and they won't close a reasoning-at-this-depth gap without training compute they can't get. If you're a research institution, your path runs through an API contract with one of two labs, and you should assume the pricing reflects that.

Where they split. The Skeptic and the Researcher are arguing about the same fact and reaching opposite places. The Skeptic says catching calculation errors proves the model is a calculator, not a creator. The Researcher says the fact that the answer key itself had to move is the whole point, no matter the sample size. That's the real fight: is "AI corrects experts" a measurement artifact from 50 hand-picked problems, or a line crossed? The second split is the Safety Lens against the Builder. The Builder wants to wire this into a research pipeline on Tuesday. The Safety Lens points out that the exact capability that makes it useful, overriding expert judgment with confidence, is the one with no off-switch when you move it to a dangerous domain.

What this hinges on is one question the story does not answer: how many of the 50 problems needed genuinely new mathematical structure, versus a clever application of known methods to an unpublished setup. If it's mostly the latter, this is a spectacular calculator and the creativity claim is marketing. If even a handful needed new structure, Clark's claim has legs. Before anyone rewrites their trust model around this, that breakdown has to exist. It doesn't yet. And at 50 problems, no statistical claim survives contact with a real sample.

Prediction: The TheoPhysEval authors will publish an expanded benchmark of at least 150 unpublished theoretical physics problems before 2027-07-01, and GPT-6-Astra's accuracy on that larger set will come in below its 98% on the original 50.

Confidence: Medium. Small hand-built benchmarks almost always regress when expanded.

Why: The 98% rests on 49 of 50 problems, a count too small for any stable estimate, and the authors already had to revise answers mid-evaluation, so they know the current set is contested and will want a bigger, cleaner version to defend the claim. When tiny benchmarks get expanded, the headline score nearly always drops, because the first batch is curated and the long tail surfaces harder cases and ambiguous grading that the small set hid. The opposite outcome, the score holding or rising on triple the problems, would require the original 50 to have been a representative sample of a domain where the model is near-perfect, and nothing about a 50-problem hand-compiled set supports that. The creativity claim will get quieter as the denominator gets honest.

Revisit by 2027-07-01: We're right if an expanded TheoPhysEval of 150 or more problems reports GPT-6-Astra below 98%. We're wrong if the expanded set reports 98% or higher, or if no expanded benchmark of that size is published by that date.

One more thing the Compute Pragmatist is right about and nobody should miss: whatever this capability is worth, it lives behind the two labs with the compute to run it, and the open-weights world is not invited to this particular party.