The AI Rankings Just Got Reshuffled

Artificial Analysis rebuilt its AI benchmark index this week, and the methodology change matters as much as the results.

The composite gap between the best US closed models and the best open-weight or Chinese models jumped from 3 points to 12. Google dropped Gemini 4 Carbon the same week and took the top spot at 59. But before you update your model strategy, look at what produced that gap.

Artificial Analysis increased its private-data share from 45% to 60% (a real improvement, stops labs from training on the test answers), dropped long-context tests, and doubled down on agentic tasks and scientific reasoning. Reweight those categories and you can produce almost any ranking you like. GPT-6 Astra went from 6th to 2nd without the model changing at all. That is the ruler getting repainted. The model did not get smarter. Haseeb Qureshi noted the new rankings "match the vibes better," which is exactly the wrong reason to trust a measurement.

Pull the index apart and the 12-point gap is mostly a science-and-research gap. Kimi 3.1 Pro scores 67% on Terminal Bench 4.0, the coding-agent benchmark, ahead of every US model. On Terminal Bench Science, it drops to 34% against GPT-6 Astra's 63%. Science is now a full quarter of the composite score. Kyle Chan at Brookings puts it plainly: Chinese models match or beat on some agentic tasks and trail badly on advanced scientific reasoning. If your workload is coding agents, document processing, support, or retrieval over your own data, the composite is misleading you toward overpaying.

The compute picture is a different kind of signal. Azure grew 48% to $33 billion, with OpenAI accounting for roughly $10 billion of that. Google Cloud nearly doubled to $30.7 billion. Both hyperscalers spent under their capex targets for the quarter, which reads less like softening demand and more like physical limits on components, power, and labor. That is a seller's market that probably holds through 2027.

The DOJ probe into NVIDIA complicates the open-model case. The investigation now spans the $13 billion Hugging Face deal, the SchedMD and RunAI acquisitions, and sales practices. Hugging Face is where most people download open weights. A distribution chokepoint under antitrust review is a real supply-chain risk for anyone building on the open ecosystem.

On Carbon specifically: it is not generally available yet, gated to cyber defenders and trusted partners. When it does open up, the pricing is interesting. At $7.76 per task, it is 23% cheaper than Anthropic's Opus 5.5 and roughly level with OpenAI. Three frontier labs within a few dollars per task of each other means model selection is now a routing decision. Google shipped Carbon two weeks after Argon, with Xiao Zhang describing launches as routine checkpoints running overnight agent loops. If that cadence holds, the eval harness you run before trusting a new model matters more than any vendor's leaderboard.

The call: When Artificial Analysis publishes its next Intelligence Index update after v5, an open-weight model (Chinese or Mistral) will rank first on at least one of the four individual category scores, even while the overall composite gap to the top US model stays in double digits. Kimi already leads the coding-agent category today. Mistral just shipped a trillion-parameter open model targeting cybersecurity. Category wins are visible now; composite wins are not. Confidence: medium.

The leaderboard reshuffle is a story about measurement. The capability story is Google's release cadence and three labs converging on the same price point. Run your own eval on your own tasks before you trust the composite number.

Artificial Analysis rebuilt the scoreboard everyone uses to pick models, and the new version tells a different story than the old one. The gap between the best US closed models and the best open-weight or Chinese models jumped from 3 points to 12. Google dropped Gemini 4 Carbon the same week and took the top spot at 59. Microsoft and Google both posted huge cloud numbers. And the DOJ probe into NVIDIA turns out to be much wider than anyone thought.

Here's the frame. None of this is hard to undo for a buyer. Model choice is a config change, not a marriage. What's actually being decided, for anyone who buys or builds with AI, is which scoreboard you trust when you tell your team "use this model," and whether the open-model-on-your-own-hardware plan still pencils out. Nothing here sets a hard deadline, except that Gemini 4 Carbon isn't generally available yet, so you can't act on it today even if you wanted to.

The Skeptic. A benchmark firm changed its own methodology and, surprise, the rankings now "match the vibes" better. That phrase, from Haseeb Qureshi, should make you nervous, not comfortable. "Matches what I already believed" is how you launder a measurement into a conclusion. Artificial Analysis moved its private-data share from 45% to 60% to stop labs training on the test answers. Good. But they also dropped long-context tests and cranked up the weight on agentic tasks, meaning multi-step jobs a model runs on its own. Reweight the categories and you can produce almost any reshuffle you like. GPT-6 Astra went from 6th to 2nd without the model changing at all. That's not the model getting smarter. That's the ruler getting repainted.

The Researcher. Look under the composite and the single number falls apart. Kimi 3.1 Pro scores 67% on Terminal Bench 4.0, a coding-agent test, leading every model. On Terminal Bench Science it scores 34% against GPT-6 Astra's 63%. Same model, nearly double the gap, depending on the task. So the "12-point open-weight gap" is really a science-and-research gap, and the new index happens to weight science as a full quarter of the score. Kyle Chan at Brookings says the plain version: Chinese models match or beat on some agentic tasks, trail badly on advanced scientific reasoning. The 3.3-months-and-closing figure from Wassan Al-Ghaib and the 12-point gap are both true and point in opposite directions. Pick your headline.

The Open-Source Advocate. The 12-point gap reads worse than it is for most buyers. Ask what you actually run. If you are doing coding agents, document processing, support, retrieval over your own data, the open models are close enough that the composite misleads you. Kimi leads on the real coding-agent test. And the supply side keeps filling in: Mistral just released the weights for Mistral Large 4, a trillion-parameter model aimed at cybersecurity, downloadable and self-hostable. The gap that's growing is in frontier scientific reasoning, which is not what most companies deploy. If your workload isn't research chemistry, a composite score that buries one strong category inside a weak-on-average number is telling you to overpay.

The Compute Pragmatist. The earnings lines matter more than the leaderboard. Azure grew 48% to $33B, and OpenAI alone is about $10B of that, a third of Azure. Google Cloud doubled to $30.7B. Both held capex guidance but spent under target for the quarter. NLW reads that as physical limits, components, power, labor, not softening demand. If he's right, and the numbers support him, this stays a seller's market. You will not negotiate a great GPU rate in 2027. The other shoe is the DOJ probe into NVIDIA, now spanning the $13B Hugging Face deal, the SchedMD and RunAI buys, sales tactics, and whether NVIDIA leaned on customers. Hugging Face is where most people download open models. A distribution chokepoint under antitrust review is a real supply-chain question for anyone betting on the open ecosystem.

The Builder. On Tuesday morning, Carbon changes nothing, because you can't call it. It's gated to cyber defenders and trusted partners under pre-release rules. What you can use is the cost line: $7.76 per task, 23% cheaper than Opus 5.5 and roughly on par with OpenAI. Two frontier labs plus Google within a few dollars of each other on price means your model choice is now a routing decision, not a loyalty one. The more interesting signal is Google's cadence. Carbon shipped two weeks after Argon. Xiao Zhang describes launches as routine checkpoints now, with overnight agent loops rewriting code on their own. If that holds, the model you standardized on last quarter is stale this quarter, and your eval harness, the test suite you run before trusting a new model, matters more than any vendor's leaderboard.

The tensions are clean. The Researcher and the Open-Source Advocate say the 12-point gap is a science artifact that doesn't touch most production work. The Skeptic says the whole reshuffle is a methodology choice that produced a predetermined conclusion. Both can be right, and together they mean the composite number is the least useful thing Artificial Analysis published. The Compute Pragmatist and the Builder disagree on where the risk sits: the Builder sees abundant, cheap, interchangeable models; the Pragmatist sees the people who sell the compute to run them hitting a wall and holding pricing power.

What this hinges on: does the US-China gap live in the specific tasks you run, or in the composite average you don't? For almost every buyer, the answer is the composite, which means the gap is mostly irrelevant to you and you should run your own eval on your own tasks before believing any single number. The council leans hard that way. The leaderboard reshuffle is a story about measurement, not capability. The capability story is Google's release speed and the fact that three labs now sit within a few dollars per task of each other.

Prediction: When Artificial Analysis publishes its next Intelligence Index update after v5 (the version released this week), an open-weight model (Chinese or Mistral) will rank first on at least one of the four individual category scores, even while the overall composite gap to the top US model stays in double digits.

Confidence: Medium — Kimi already leads the coding-agent test today; category wins are visible, composite wins are not.

Why: The episode shows Kimi 3.1 Pro already scoring 67% on Terminal Bench 4.0, the coding-agent benchmark, ahead of every US model, while trailing badly on the science category that v5 now weights as a full quarter of the total. That split is the whole story: open models win specific tasks and lose the average. Category leaderboards expose that, composites hide it, and Artificial Analysis publishes both. Mistral just shipped a trillion-parameter open model aimed at cybersecurity, adding another contender for a single category crown. The opposite outcome, open models losing every individual category, would require the US labs to simultaneously retake coding-agent tasks they currently trail on, which nothing in this release suggests is happening.

Revisit by 2027-05-04: We're right if the next Artificial Analysis Intelligence Index release lists any open-weight model first in at least one of the four category scores. We're wrong if US closed models sweep all four categories.

Run your own eval on your own tasks before you trust any of these numbers. The leaderboard tells you what Artificial Analysis chose to weight. Your workload tells you what to buy.