Refacto AI

Podcast episode

Why AI Washing Won’t Work Much Longer

evals inference model-pricing open-weights

Nathaniel Whittemore's episode covers two connected threads: Alibaba dropping Qwen 3.8 Max at $2/$6 per million tokens with open-weights access, and a New York Times op-ed arguing that AI washing (overclaiming AI impact to investors and buyers) is running out of road as enterprise buyers get more sophisticated.

The pricing looks aggressive until you factor in retries. Independent researcher Pavel Horan found Qwen fixed 19 of 105 bugs at roughly $31 per task, needing five attempts, while GPT-5.6 Luna fixed 33 for $1.80. Cheap per-token means nothing if the model flails and charges you five times. Separately, Google DeepMind's Jeet Singh told a UC Berkeley audience that current AI revenues don't cover hyperscaler capex, framing the spend as a bet on recursive self-improvement (AI systems that iteratively improve themselves). Subsidized pricing is not real pricing.

The open-weight release matters more than any benchmark. A frontier-adjacent model you can run in your own environment permanently changes what closed labs can charge. Route a batch job to Qwen Tuesday; measure cost-per-completed-task, not cost-per-token.

Full analysis

The theme running through this episode is that enterprises are getting harder to fool about AI. Two claims sit at the center for anyone shipping AI into production: Alibaba's Qwen 3.8 Max landing at $2/$6 per million tokens with open weights, and a New York Times op-ed arguing that "AI washing" (claiming more AI impact than exists) is about to stop paying dividends. Both are easy to reverse if you're wrong. Trying Qwen on one batch workload costs you an afternoon, not a re-architecture. That makes this a Type 2 decision: act fast, verify cheap, don't hold a strategy offsite about it.

What's actually being decided isn't "should we use Qwen." It's whether the cheap open-weight tier is now good enough to route real agentic work to, and whether your own AI claims survive the sophistication your buyers just picked up. The forcing function is real: full Qwen weights drop the following week, and the independent benchmarks are already in conflict.

The Skeptic. Self-reported benchmarks are a press release with a chart. Alibaba claims state-of-the-art on OS World Verified (a test of AI driving a computer like a person), and I'd bet nothing until someone who doesn't sell the model runs it. The independent numbers already disagree: Pavel Horan found Qwen fixed 19 of 105 bugs at roughly $31, needing five retries, while GPT 5.6 Luna fixed 33 for a buck eighty. That's not a pricing win, that's a model that burns your quota flailing. Cheap per-token means nothing if you pay for the tokens five times. For a PM: the sticker price is low, but the bill depends on how many tries it takes to get a right answer.

The Researcher. The 2.4 trillion parameter headline is close to meaningless without the active-parameter count, and Alibaba isn't leading with it. What matters is the gap between the self-reported placement (between GPT-4.5 Sol and Fable 5 on terminal bench) and the independent read that put it four points behind Kimi K3. Four points on a bug-fixing benchmark is the difference between shipping and not. Horan's cost-per-fixed-bug numbers are the metric that counts, because they fold accuracy and retries into one figure. On that axis Qwen looks expensive despite the cheap tokens. Wait for the weights, run your own eval on your own task distribution.

The Open-Source Advocate. Set the benchmark squabble aside, because the structural move is what counts. Alibaba is open-sourcing weights for a Max-class model for the first time. That's a frontier-adjacent model you can run in your own environment, which is the exact thing Palantir's Alex Karp is selling as "AI sovereignty" without letting you own anything. Karp's whole pitch is that enterprises want control over their data and prompts. Open weights deliver that control directly, no Palantir seat license required. For the reader: this is the difference between renting intelligence and owning a copy of it. Even if Qwen 3.8 lands slightly behind Kimi K3, the pressure it puts on closed-model pricing is the story. Claude Opus at $25 per million output tokens has to explain a 4x premium now.

The Compute Pragmatist. Jeet Singh at Google DeepMind said the quiet part into a UC Berkeley microphone: current AI revenues "don't sustain the capital expenditures we're making so far," and there's a "danger we could hit an AI air pocket." Then he justified the spend as a bet on recursive self-improvement, AI systems that iteratively improve themselves. Translation for anyone modeling their inference bill: the hyperscalers are committed to a long compute runway regardless of near-term cash. That's good news for supply and eventually for prices. It's bad news if you're building on the assumption that today's frontier pricing reflects real economics. It doesn't. It reflects a subsidized land grab, and subsidies get repriced.

The Builder. What do I actually do Tuesday? I don't rip out Claude for Qwen on anything customer-facing. Five retries and stability issues mean an angry on-call engineer at 3 AM. But I absolutely spin up Qwen against a batch job with no latency SLO and generous retry budget, measure cost-per-completed-task, not cost-per-token, and see if the math beats what I'm paying now. And I take the Claude DNA-database exploit seriously as a work item, not a headline. Researchers used Claude to find and exploit a tamper vulnerability in 1995-era forensic software in about 45 minutes. If your stack leans on any legacy dependency, that same capability is now pointed at you. Run an AI-assisted audit before someone else does.

The disagreements worth sitting with: the Open-Source Advocate sees a pricing earthquake where the Skeptic and Researcher see a model that isn't reliable enough to route real work to yet. Both are right, on different timelines. Qwen 3.8 today is a batch-job experiment. Qwen's existence as an open-weight Max-class model is a permanent change to what closed labs can charge. The second disagreement cuts deeper: the Compute Pragmatist hears Sikhan admit the capex doesn't pencil out, while Karp is on the same episode declaring Palantir the only company turning tokens into "actual economic value." One of them is describing an air pocket and the other is selling parachutes. They can't both be describing a healthy market.

What this actually hinges on: does the cheap open-weight tier close the reliability gap on agentic tasks before your closed-model contract renews? If yes, the pricing pressure is real and you should be building routing that can swap tiers. If the retry problem persists, Qwen stays a batch curiosity and Claude keeps its premium. The council leans toward the pricing pressure being real and the current model not being ready. Both can hold. Before committing anything, run Qwen 3.8 on your own agentic eval the week the weights land, measure cost-per-completed-task including retries, and put a tier-swap clause or a short renewal term in any closed-model contract you sign this quarter.

On the "AI washing" thesis, the op-ed by former Lululemon CIO Julie Averell has one number that grounds it: of the 97,000 U.S. job cuts in May with 40% blamed on AI, about a third of those roles were quietly re-hired. That's the sophistication shift host Nathaniel Whittemore is pointing at. Boards stop rewarding the AI headline once the re-hires show up in the next quarter's costs.

Prediction: When Alibaba releases full Qwen 3.8 Max weights and independent testers run agentic coding benchmarks (SWE-bench, terminal bench, OS World Verified) within roughly 30 days of the drop, at least one major independent tester will rank it behind Kimi K3 on cost-per-completed-task, not just raw score.

Confidence: Medium. The early independent run by Pavel Horan already put it four points and several dollars behind on cost-adjusted task completion.

Why: The one independent data point in this episode, Pavel Horan's run, already had Qwen fixing 19 of 105 bugs at roughly $31 with five retries, while cheaper models fixed more for a fraction of the cost. The cheap $2/$6 token price is undercut by the retry burn, and reliability issues rarely vanish between a preview and a weights release. The opposite outcome, Qwen topping the independent agentic leaderboards on real cost-adjusted terms, would require the stability problems to disappear in a week, which self-reported benchmarks give no reason to expect.

Revisit by 2026-09-15: We're right if at least one independent agentic-coding benchmark ranks Qwen 3.8 Max behind Kimi K3 on cost-per-completed-task after the weights ship. We're wrong if independent testers put Qwen ahead of Kimi K3 on both accuracy and cost-adjusted throughput.

Comments