Refacto AI

Industry story

GPT-6 Astra Sets Records on Epoch Benchmarks, First to Crack Erdős Math Problems

agents evals inference model-pricing

OpenAI's GPT-6 Astra, released September 3, 2026, was pre-release evaluated by Epoch AI and set new top scores across multiple benchmarks, taking the highest position among the 247 models tracked on the Epoch Capabilities Index. On mathematics, Astra is the first AI to solve any problems on Epoch's new FrontierMath Erdős benchmark — a set of 68 unsolved problems posed by mathematician Paul Erdős, formalized in the proof-verification language Lean — cracking 2 out of 68. It also scored 98% on FrontierMath Tier 4, effectively saturating that benchmark. These results mark a notable capability step for frontier AI in formal mathematical reasoning.

Analysis

Showing the shorter version.

GPT-6 Astra: Real Math Breakthrough or Pre-Release Marketing?

OpenAI's GPT-6 Astra, released September 3, 2026, topped every benchmark Epoch AI tracks and became the first model to solve any of the 68 unsolved Erdős math problems, cracking 2 of them in Lean. Lean is a proof-checking language that machine-verifies correctness and rejects anything wrong, so whatever Astra produced is correct by construction. That matters: it's categorically different from a chatbot confidently stating false math. Astra also hit 98% on FrontierMath Tier 4, which effectively retires that benchmark as a discriminator between top models.

The central question is whether frontier models have crossed from reproducing human-solved math to producing new, verified math that humans could not. If yes, the value of a frontier model tilts toward domains where correctness can be machine-checked: formal proofs, code verification, hard scientific computation.

The skeptic's case is reasonable but fixable. Two out of 68 is a 3% hit rate on problems chosen precisely because nobody could solve them. And the Tier 4 result needs a caveat: Epoch built that test before Astra existed, but keeping training data clean of it is hard to prove, especially when OpenAI granted Epoch the pre-release access used to run the evaluation. The skeptic's objection dies the moment OpenAI or Epoch publishes the two proofs. A Lean proof is checkable by anyone with the software. Withholding it keeps this a marketing narrative instead of a scientific result. Until the proofs are public and independently confirmed as non-trivial, the skeptic wins by default.

The cost shape is different from ordinary AI. Solving hard math almost certainly required heavy inference-time compute: extended thinking, tree search over proof states, verification-guided generation. Expect hard-math queries to run 10 to 100 times the price of a normal completion. The per-token price tells you nothing useful here. Track cost per solved problem. And benchmark conditions are not production conditions: latency on deep proof states, context behavior as a proof runs long, and API pricing at real volume will break before the model's math ability does.

The second-order consequence worth taking seriously: self-verifying math is the exact ingredient that makes autonomous research agents viable. The same capability that makes a result trustworthy makes the downstream agent harder to keep a human on top of. Lean proofs being auditable does not make Astra's ordinary reasoning auditable. The verifiable part is a thin slice of what the model actually does.

The call: OpenAI or Epoch will publish both Lean proofs, and independent mathematicians will confirm them as valid and non-trivial, by December 13, 2026. Confidence: medium. A Lean proof costs nothing to release and settles every contamination doubt. The marketing value of "first to crack Erdős" depends entirely on the result surviving scrutiny, and the cheapest way to bank it is releasing two files that verify on anyone's machine. If the proofs stay unpublished past three months, that silence is the answer.

Also covered this issue

Comments