SemiAnalysis Launches AgentX 1.0: First Open-Source Agentic Inference Benchmark

Full analysis

SemiAnalysis, Dylan Patel's shop, just dropped AgentX 1.0: an open-source benchmark that measures how inference actually behaves under agentic coding workloads. Built from $3M in real Claude Code and Codex traces, 610 billion tokens, 1M-token context. The claim underneath it matters more than the benchmark itself: agentic spend has overtaken chat spend in OpenAI's enterprise segment, and nobody's inference stack was tuned for that traffic shape.

What's being decided for the reader: whether to re-cut your inference capacity planning and hardware bets around agent-shaped traffic, or keep running the synthetic fixed-sequence benchmarks you already have. Type 2, mostly reversible. You can run AgentX against your stack this week and roll back. The forcing function is real: if tier-1 labs are already consuming the public API for capacity planning, your competitors are re-pricing their inference bill against this while you're still benchmarking against MMLU-era fiction.

The Skeptic. Two providers, both frontier labs, both coding. That's the whole trace corpus. If you serve open-weight models, non-coding agents, or anything vertical, AgentX describes someone else's traffic. The "first open-source agentic benchmark" line is marketing doing work prior efforts existed, they just weren't Dylan Patel's. And the 70 PRs cut both ways: when a benchmark starts driving code changes in vLLM and SGLang, you get optimizations tuned to the benchmark, not to your production. The April 2026 OpenAI inflection is one segment at one lab. Real, but one data point is not a market law. For the PM: this measures how Anthropic and OpenAI's coding agents run, not necessarily how yours will.

The Compute Pragmatist. This is the part that moves money. Prefix reuse at 1M context is a memory-bandwidth and KV-cache problem, not a FLOP problem. That quietly rewrites the GPU buying case. If you procured on training-era FLOP-per-dollar math, you optimized the wrong axis for this workload. AMD's MI300X, with its fatter HBM pool, suddenly reads better for long-context agents than the FLOP leaderboards ever suggested against NVIDIA. Sub-agent bursts mean spiky, unpredictable batch sizes, which is a scheduler problem that rewards disaggregated prefill and decode. For the PM: the benchmark says the bottleneck moved from raw math to memory, and that changes which chip you rent.

The Researcher. 610B tokens of live production traces beats synthetic scaffolding, full stop. This is the first agentic benchmark that earns the realism claim, and Apache 2.0 means the community can audit it before it Goodharts itself into uselessness. But watch the composition. $3M in Claude Code traces over-represents Anthropic-style loops, lots of sub-agents, heavy tool-calling. That's a real shape, but one shape among several. The 70 upstream PRs are the strong signal: a benchmark that changes the systems it measures is doing its job, provided the corpus stays open enough that nobody quietly overfits the serving frameworks to one lab's traffic. For the PM: it's built from what really ran in production, not a lab test, which is rare and good.

The Enterprise Buyer. Here's what a CTO actually does with this. Your inference contract, whether it's a per-token deal with a provider or a GPU reservation, was priced against a traffic mix that no longer exists. Agentic workloads burn 10-100x the tokens of a chat turn because they loop, call tools, and spawn sub-agents. If AgentX lets you model your true token burn under agent traffic before you renew, that's leverage at the negotiating table. The risk is buying the "this is how production works" framing wholesale and re-architecting around coding-agent patterns when your revenue workflow is a support agent or a data pipeline. For the PM: the useful thing here is a way to estimate what agents will actually cost you before the bill arrives.

Where they part ways. The Compute Pragmatist wants you to re-cut hardware bets now, because the memory-bandwidth story is real regardless of whose traces you use. The Skeptic says the corpus is too narrow to bet chips on. That tension is the whole decision: the workload physics AgentX exposes (KV cache, prefix reuse, burst scheduling) generalize even if the specific traffic mix doesn't. Prefix reuse punishing HBM is true for any long-context agent, coding or not. The composition bias hits the tuning. The physics hold either way.

The second split: the Researcher trusts the open license to keep the benchmark honest; the Safety angle nobody's raising is that optimizing every serving framework for autonomous multi-turn tool-calling normalizes the exact workload where human oversight is thinnest. Both true. Neither changes your Tuesday.

What it hinges on. One belief: does the workload shape AgentX captures, long context, high prefix reuse, sub-agent bursts, generalize beyond coding agents? If yes, the hardware and capacity conclusions hold even though the trace corpus is narrow. If no, you've tuned for Anthropic's traffic. The way to settle it is not to argue about the corpus. Run AgentX against your own traffic, then diff your real KV-cache hit rate and batch-size distribution against the benchmark's. If they rhyme, the physics generalize and you act on it. If they diverge hard, you have a coding-agent benchmark, not yours.

The council leans toward: the physics are real, the corpus is narrow, and the interesting consequence isn't the benchmark at all. It's that agentic traffic broke the hardware assumptions the last three years of GPU buying were built on.

Prediction: By the end of Q1 2027, at least one of vLLM or SGLang will ship disaggregated prefill/decode serving as a default or first-class configuration, with AgentX-style agentic workloads cited as the motivating case in the release notes or docs.

Confidence: Medium. The 70 PRs and burst-scheduling problem point straight at disaggregation, but timing across open-source projects is loose.

Why: AgentX exposes that agentic traffic is spiky and prefix-heavy, which makes the old approach of running prefill and decode on the same GPU inefficient, because a sudden sub-agent burst starves the decode step that keeps latency low. The known fix is disaggregation: split prefill and decode onto separate pools so a burst on one doesn't stall the other, and it's already landing as experimental in these frameworks. When a benchmark is driving 70 upstream PRs and tier-1 labs are consuming it for capacity planning, the frameworks follow the traffic the benchmark exposes, because that's where the measured wins are. The opposite outcome, disaggregation staying an experimental flag, would require the maintainers to ignore the one workload the benchmark proves is now the majority of production traffic, which cuts against how these projects have historically prioritized.

Revisit by 2027-03-31: We're right if vLLM or SGLang documents disaggregated prefill/decode as a default or first-class path with agentic/long-context workloads named as the driver. We're wrong if both keep it experimental or opt-in with no such framing.

The narrower corpus is the caveat. The physics of prefix reuse punishing HBM bandwidth hold whether the agent writes code or files support tickets, and that's what re-prices the chip decision.