The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

Full analysis

Two Baseten engineers, Philip Kiely and Ali Taha, spent two hours walking through what it actually takes to serve open-weight LLMs in production, and the headline for anyone running inference is simple: you are probably leaving a 4-6x speedup on the floor. Same weights, different provider, and the spread on OpenRouter runs 4-6x between the worst and best. That gap is your money.

Reversibility: Type 2, mostly. Swapping inference providers or turning on speculative decoding for one workload is cheap to undo. The Type 1 buried inside is architectural: whether you build for closed-loop trace collection and continuous fine-tuning from day one, because retrofitting that later is a rewrite.

What's actually being decided: not "which model," but "do we treat inference as a solved commodity we rent, or as an engineering surface we own." The episode argues the latter still has enormous gains left.

Forcing function: none hard. But NVFP4 on Blackwell, GLM-5.2's sparse attention, and Rubin's design choices mean the stack you tuned last year is already stale.

The Skeptic. Most of the 10x is real but it is not free, and it is not yours unless you run dedicated. Speculative decoding at 2x requires a draft model trained on your traffic. Prefill-decode disaggregation needs separate GPU pools. The KL-divergence quantization trick came from a 45-page intern paper, not a config flag. Every one of these gains is an engineering team-quarter, not a checkbox. And the token-loop collapse bug, where identical weights on different clusters produce different reliability, means you cannot even trust a checkpoint that worked last week on new hardware. To the PM: the cheap magic numbers everyone quotes assume a team that does not exist at most companies.

The Researcher. The genuinely new claim is that quantization is not monotonically lossy. Baseten's Joshua showed that picking which layers to quantize by predicted error direction, measured via KL divergence (how far the compressed model's output probabilities drift from the full model's), yields a model 20% more compressed and more faithful, because errors cancel across layers. That contradicts the naive "less precision equals less quality" intuition everyone ships on. Kiely's framing is right: when the field brags about 1% gains, inference is solved. Right now people publish 200%. To the PM: we are still in the era where careful engineering buys you double, not decimal points.

The Open-Source Advocate. This is the open-weight operator's playbook, and it is the whole point. The tricks only pay off because you hold the weights: custom speculators, custom quantization, Haley grafting Kimi's vision encoder onto GLM-5.2 by freezing both and training a few-million-parameter projector to 56% on MMLU Pro. You cannot do any of that on a closed API. But the gap is real too. Open-source video (Wan 2.2, 20B) is nowhere near Veo or Kling, because full attention over 35,000 tokens for five seconds of 480p is O(n²) and intractable. To the PM: on text and agents, open weights plus engineering closes the gap. On video, it does not, yet.

The Compute Pragmatist. The economics are blunt. At millions of tokens an hour, hourly dedicated billing beats per-token API, and only dedicated unlocks the 2x speculator and the custom quant. So the crossover is a volume question you can compute today. But Ali Taha names the real ceiling, and it is not GPU design. Faster NICs (the network chips moving data between servers) would give nearly 100x on disaggregated serving across nodes. Inter-node KV cache transfer is the blocker. Meanwhile Rubin is built to kill mega-kernels, so betting a team on CUDA kernel tuning is betting against NVIDIA's roadmap. To the PM: your leverage is moving to orchestration, not thread-level control.

The Builder. What ships Tuesday: measure your current provider spread before you build anything. If you are on a shared endpoint at high volume, a dedicated deployment with a traffic-trained speculator is the fastest real win, and it needs a rollback plan because the collapse bugs are hardware-specific. The bigger architectural call is Kiely's continual-learning loop: inference, learn from inference, post-train, A/B, better model. He puts it at a few months to a couple years. If you are building agents, wire trace collection in now, because the traces are your highest-signal training data and you cannot recover them retroactively.

Where the council splits:

The Researcher and the Skeptic disagree on whether the quantization result is a tool or a trophy. A 45-page paper proving errors cancel is beautiful; a team that can reproduce it per-model, per-cluster is rare. The gain is real and mostly unclaimable without serious staff.

The Open-Source Advocate and the Compute Pragmatist part on where the ceiling sits. One sees weight access as the unlock; the other says the unlock is NIC bandwidth nobody at your company controls. Both are right, which means your gains are capped by hardware you rent, not code you write.

The Builder and the Skeptic split on the continual-learning loop. Ship the trace pipeline now versus wait until the pattern stabilizes. Building it early is cheap insurance; building the full loop early is chasing a moving target.

What it hinges on: whether your token volume clears the dedicated-deployment crossover. Below it, this whole episode is interesting reading and you rent from whoever is fastest this month. Above it, owning the inference stack buys 4-6x, and the traces buy you a better model. The verify step is dull and mandatory: benchmark your live provider against Artificial Analysis, measure your actual tokens-per-hour, and load-test candidate models per cluster before trusting a checkpoint.

The one belief to interrogate before committing a team: do you believe the continual-learning loop lands in months, as Kiely says, or years. Architect the trace collection either way. Delay the closed loop until the tooling settles.

Prediction: By the end of Q2 2027, no major open-weight video model (the Wan line or a successor at comparable open scale) will match Veo or Kling on standard quality benchmarks for clips longer than 10 seconds.

Confidence: Medium. The O(n²) attention wall is a hard compute constraint, not a tuning gap.

Why: The episode pins open-source video's deficit on a specific mechanism: full attention over 35,000 tokens for five seconds of 480p at 16fps, which grows with the square of length and becomes intractable past short clips. Ali Taha's own escape hatches are either autoregressive video, which he calls "terrible quality" today, or an "insane leap in compute" for full attention over millions of tokens. Neither is a quarters-away fix, and long-form stitching of 7-second chunks drifts to black screen. The opposite outcome would require an architectural breakthrough in autoregressive video quality that no open lab is close to shipping, so the gap holding is the safer call.

Revisit by 2027-06-30: We're right if the top open-weight video model still trails Veo/Kling on human-preference or standard video benchmarks for 10-second-plus clips. We're wrong if an open model at Wan-comparable scale matches or beats them on long-form quality.