Industry story
GPT-6 Astra Sets Records on Epoch Benchmarks, First to Crack Erdős Math Problems
agents evals inference model-pricing
OpenAI's GPT-6 Astra, released September 3, 2026, was pre-release evaluated by Epoch AI and set new top scores across multiple benchmarks, taking the highest position among the 247 models tracked on the Epoch Capabilities Index. On mathematics, Astra is the first AI to solve any problems on Epoch's new FrontierMath Erdős benchmark — a set of 68 unsolved problems posed by mathematician Paul Erdős, formalized in the proof-verification language Lean — cracking 2 out of 68. It also scored 98% on FrontierMath Tier 4, effectively saturating that benchmark. These results mark a notable capability step for frontier AI in formal mathematical reasoning.
Analysis
Showing the shorter version.
GPT-6 Astra: Real Math Breakthrough or Pre-Release Marketing?
OpenAI's GPT-6 Astra, released September 3, 2026, topped every benchmark Epoch AI tracks and became the first model to solve any of the 68 unsolved Erdős math problems, cracking 2 of them in Lean. Lean is a proof-checking language that machine-verifies correctness and rejects anything wrong, so whatever Astra produced is correct by construction. That matters: it's categorically different from a chatbot confidently stating false math. Astra also hit 98% on FrontierMath Tier 4, which effectively retires that benchmark as a discriminator between top models.
The central question is whether frontier models have crossed from reproducing human-solved math to producing new, verified math that humans could not. If yes, the value of a frontier model tilts toward domains where correctness can be machine-checked: formal proofs, code verification, hard scientific computation.
The skeptic's case is reasonable but fixable. Two out of 68 is a 3% hit rate on problems chosen precisely because nobody could solve them. And the Tier 4 result needs a caveat: Epoch built that test before Astra existed, but keeping training data clean of it is hard to prove, especially when OpenAI granted Epoch the pre-release access used to run the evaluation. The skeptic's objection dies the moment OpenAI or Epoch publishes the two proofs. A Lean proof is checkable by anyone with the software. Withholding it keeps this a marketing narrative instead of a scientific result. Until the proofs are public and independently confirmed as non-trivial, the skeptic wins by default.
The cost shape is different from ordinary AI. Solving hard math almost certainly required heavy inference-time compute: extended thinking, tree search over proof states, verification-guided generation. Expect hard-math queries to run 10 to 100 times the price of a normal completion. The per-token price tells you nothing useful here. Track cost per solved problem. And benchmark conditions are not production conditions: latency on deep proof states, context behavior as a proof runs long, and API pricing at real volume will break before the model's math ability does.
The second-order consequence worth taking seriously: self-verifying math is the exact ingredient that makes autonomous research agents viable. The same capability that makes a result trustworthy makes the downstream agent harder to keep a human on top of. Lean proofs being auditable does not make Astra's ordinary reasoning auditable. The verifiable part is a thin slice of what the model actually does.
The call: OpenAI or Epoch will publish both Lean proofs, and independent mathematicians will confirm them as valid and non-trivial, by December 13, 2026. Confidence: medium. A Lean proof costs nothing to release and settles every contamination doubt. The marketing value of "first to crack Erdős" depends entirely on the result surviving scrutiny, and the cheapest way to bank it is releasing two files that verify on anyone's machine. If the proofs stay unpublished past three months, that silence is the answer.
OpenAI's GPT-6 Astra, out September 3, 2026, topped every benchmark Epoch AI tracks and became the first model to solve any of the 68 unsolved Erdős math problems, cracking 2 of them in Lean, a language that machine-checks a proof and rejects it if it's wrong. It also hit 98% on FrontierMath Tier 4, which means that test is now useless as a way to tell top models apart. This is a capability claim, and the question for anyone who buys or builds with AI is: what's actually new, and what's marketing sitting on top of pre-release access.
How hard is this to undo? Nothing to undo. Nobody has to commit to anything today. Astra is a new option, not a fork in the road. That means less agonizing and faster testing.
What's actually being decided: not "is Astra impressive." It's whether frontier models have crossed from doing math that humans already solved to producing new, verified math humans could not. If yes, the value of a model shifts from "answers fast" to "answers correctly on problems where correctness can be checked." That repricing is what changes the buying calculus.
What sets the deadline: nothing external. No contract, no shutdown. The only clock is competitive: Google DeepMind and Anthropic will answer, and Epoch will need a harder test.
The Skeptic. Two out of 68 is a 3% hit rate on problems chosen precisely because nobody could solve them. "Cracks Erdős" is a headline; "fails 97% of the time" is the same fact. And Tier 4 at 98% needs a caveat: that test was built before Astra existed, and keeping training data clean of it is hard to prove, especially when OpenAI handed Epoch the pre-release access. The 247-model ranking is only worth something if the ruler is right, and OpenAI helped pick the ruler. Show me the two proofs. Have three mathematicians with no stake confirm they're non-trivial and not a known result restated. Then we talk.
The Researcher. Two Lean-verified Erdős proofs are not pattern-matching. Lean rejects a wrong proof outright, so whatever Astra produced is correct by construction. That is different in kind from a chatbot confidently stating false math. Tier 4 saturating means it is retired as a discriminator, and the field needs harder tests now. The interesting scientific question is whether these two solutions expose structure in the other 66, or whether Astra found a proof-search shortcut humans don't use. Either answer matters. And the fix for the Skeptic is trivial: publish the proofs. A Lean proof is checkable by anyone with the software. Withholding it is the only thing that would keep this a narrative instead of a result.
The Safety Lens. Machine-checkable truth cuts both ways. Interpretability researchers get a clean substrate: a reasoning chain that either verifies or doesn't, no hand-waving. But the same capability feeds research agents that can check their own work and keep going without a human in the loop. Self-verification is the ingredient that turns a smart model into one that runs long autonomous chains. Two Erdős solutions say Astra has internalized proof-search strategies that were human-only until now. And the comfort is false: Lean proofs being auditable does not make Astra's ordinary reasoning auditable. The verifiable part is a thin slice of what the model does.
The Compute Pragmatist. Saturating Tier 4 and generating Erdős proofs almost certainly needed heavy inference-time compute: extended thinking, tree search over proof states, verification-guided generation. That is a different cost shape than chat. Expect hard-math queries to run 10 to 100 times the price of a normal completion, because the model searches and checks hundreds of candidate paths before returning an answer. The infrastructure consequence: the Lean checker now has to live inside the inference loop, not bolted on after. Holding long proof-state context at batch will strain GPU memory first. For a buyer, the sticker price per token tells you nothing about the cost of one solved problem.
The Builder. On Tuesday, this matters to anyone shipping scientific software, theorem provers, or code-verification pipelines. Test Astra on your hardest edge cases this quarter. But a benchmark harness is not a researcher's desk. Lean proof generation scored in a controlled run is not the same as interactive proof help while a human iterates. Latency on long proof states, context behavior when the proof runs deep, and API pricing at real volume will break before the model's math ability does. Benchmark day and production day are different days.
Where they split. The Researcher and the Skeptic disagree on one concrete thing: are the two proofs real, novel math or a clever shortcut on a problem that was more tractable than its "unsolved" label suggested. This is fully resolvable. Lean proofs are public-checkable, so the disagreement dies the moment OpenAI or Epoch posts them. Until they do, the Skeptic wins by default.
The second split is Safety versus everyone bullish on the capability. Self-verifying math is the exact ingredient that makes autonomous research agents work. The thing that makes the result trustworthy is the thing that makes the downstream agents harder to keep a human on top of.
What it hinges on. One fact: are the proofs published and independently verified as non-trivial. If yes, this is a genuine capability step and the value of frontier models tilts toward verifiable-answer domains: math, formal code, anything you can machine-check. If the proofs stay unpublished, "first to crack Erdős" is a pre-release marketing line resting on access OpenAI granted. Before anyone builds a workflow on Astra's math, run your own hardest verifiable problems through it at production volume and track the cost per solved problem. The per-token price is the wrong number to watch.
Prediction: OpenAI or Epoch AI will publish the two GPT-6 Astra Lean proofs for the FrontierMath Erdős problems, and independent mathematicians will confirm both as valid, by December 13, 2026 (roughly three months after the September 3 release and the next Epoch Brief cycle).
Confidence: Medium. Lean proofs are trivially checkable by anyone with the software, and the claim loses most of its value if withheld.
Why: A Lean proof is machine-checkable by anyone with the software, so publishing it costs OpenAI nothing and settles every contamination and cherry-picking doubt the Skeptic raised. The whole marketing value of "first to crack Erdős" depends on the result surviving scrutiny, and the cheapest way to bank that value is to release two files that verify on anyone's machine. The opposite outcome, keeping the proofs private, is the less likely path because it hands every competitor a reason to call the result unverified, and OpenAI has more to gain from a clean win than from guarding a proof that Lean already confirmed is correct. The risk to the call is timing: they may sit on it until a paper or a harder-benchmark launch, pushing publication past the window.
Revisit by 2026-12-13: We're right if both Erdős proofs are public and at least one independent mathematician confirms they're valid and non-trivial. We're wrong if the proofs remain unpublished or an independent check finds either one trivial or a restated known result.
Speed is the evidence either way. A real result gets published fast because publishing wins the argument. Silence past three months means the argument is the point.
Also covered this issue
-
Dario Amodei Calls for AI Pacing, Embedded Safety Evaluators
zvi-vase
Anthropic embedded outside safety reviewers in its training process, but only promised to report problems internally, leaving your company unable to verify what they actually found.
-
OpenAI Blog Post Confirms Automation of AI R&D Itself
zvi-vase
Faster AI model updates could break your production systems and integrations unless you build automated tests to catch when upstream models change behavior under your workloads.
Comments