Podcast episode

The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

by Ken Rona ·August 6, 2026 ·Latent Space

cost-compression inference open-weights orchestration

TL;DR

A two-hour deep dive with Baseten's Philip Kiely and Ali Taha — authors of the "Inference Engineering" book — covering the full stack of production LLM serving: from cache-aware routing and disaggregated prefill/decode, to quantization error cancellation, speculative decoding, GPU parallelism, and the convergence of training and inference. Essential listening for anyone operating or building on open-weight model infrastructure.

What was covered

End-to-end anatomy of a 200K-token request: cache-aware routing to find replicas with warm KV cache (key-value cache — stored intermediate computations that avoid reprocessing repeated tokens), disaggregated prefill (input processing) and decode (token generation) across separate GPU pools, and speculative decoding via a small draft model that proposes tokens the large model then verifies in one pass.
What "day-zero" model support actually requires: beyond getting a token out via vLLM/SGLang, production readiness means re-quantizing to NVFP4 for Blackwell compatibility, training a custom speculative decoder on representative traffic (coding, agentic), and standing up infrastructure — each step taking hours to days. DeepSeek models cited as most challenging due to novel architecture each release; GLM-5.2 introduced sparse attention (DSA) borrowed from DeepSeek.
Grafting Kimi's vision encoder onto GLM-5.2: Baseten engineer Haley froze both the Kimi vision encoder and GLM language model weights, then trained only a lightweight projector (~millions of parameters) to bridge them — reaching 56% on MMLU Pro without degrading the base LLM's text performance.
Quantization error cancellation research: Baseten intern Joshua produced a 45-page paper (originally 72 pages) showing that selecting which layers to quantize based on predicted error directionality — measured via KL divergence between quantized and full-precision logit distributions — can yield a model that is 20% more quantized than NVIDIA's default while achieving better fidelity, because errors cancel across layers.
Inference speedup stacking toward 10×: BF16 → NVFP4 (~2×), speculative decoding (~2×), prefill-decode disaggregation (~2×), plus runtime/kernel improvements (double-digit %). Combined, a 1T-parameter model baseline of 30–40 tokens/second can reach 300–400 tokens/second. Real-world spreads on OpenRouter/Artificial Analysis often show 4–6× between worst and best providers.
NVIDIA Dynamo, Rubin, and the shift to systems-level inference: Dynamo positioned as a KV-cache routing/movement toolkit (not a drop-in optimizer). Rubin GPU reportedly designed to reduce the value of mega-kernel approaches (fusing many GPU kernels into one), pushing inference engineering toward infrastructure orchestration rather than CUDA kernel tuning. Ali argued GPUs are trending toward "programmable ASICs."
Video diffusion vs. autoregressive video: Open-source video (Wan 2.2, 20B params) far behind closed models (Veo, Kling). Root cause: full attention over 35,000 tokens for just 5 seconds of 480p video at 16fps makes O(n²) attention intractable beyond short clips. Autoregressive video would enable streaming and spec-dec-style optimizations, but current autoregressive video quality is poor. Long-form generation via stitching 7-second chunks suffers quality drift to black screen.
Training ↔ inference convergence: Inference-for-training: slow rollouts (vLLM-based) bottleneck RL training pipelines. Training-for-inference: quantization-aware distillation (NVIDIA paper) trains FP4 models by distilling from full-precision logits. Live example: GLM-5.2 used to write and optimize its own SGLang GPU kernels in a closed loop at Baseten — the model profiled its own inference, identified bottlenecks, wrote replacement kernels, and re-profiled.

Notable claims & predictions

Philip Kiely: "When we publish optimizations, it's 20%, it's 100%, it's 200%. You'll know inference is pretty much solved when researchers start publishing about how they got 1% faster at something." — frames current inference as analogous to early quantitative finance, with large gains still available.
Ali Taha: "Faster network chip communications [NICs] would give almost 100× speedup in disaggregated serving between nodes — it's the real blocker, not the stuff you can rearrange within the existing GPU design." — identifies inter-node KV cache transfer latency as the fundamental ceiling.
Ali Taha: "The GPU is moving more towards being an ASIC — you're orchestrating what happens but not controlling it thread by thread. Rubin kills mega kernels; the entire research field won't be continued." — bearish call on mega-kernel research as a viable production path.
Ali Taha on video: "If you want Hollywood-level movies from video models, they have to be autoregressive to exceed the 5-second frame — or there has to be some insane leap in compute for full attention over millions of tokens." — current open-source autoregressive video models described as "terrible quality."
Philip Kiely on continual learning loops: "Within a few months to a couple years, leading agent builders will have continuous improvement loops in production — inference, learning from inference, post-training the model, A/B test, better model, better product." — near-term prediction for production deployment patterns.
Ali Taha on KV cache as the path to continual learning: "KV cache compaction — making KV almost infinite without losing knowledge — solves continual learning where weight editing fails, because weight edits only fix one-hop knowledge and don't generalize to multi-hop reasoning."

Why this matters for AI operators

Cost optimization decision point: The dedicated-deployment vs. shared-API crossover is clearer than commonly understood — at millions of tokens/hour, hourly billing is materially cheaper, and dedicated deployments unlock traffic-specific speculators (2× decode speed) and custom quantization configs that shared endpoints cannot offer.
Inference stack fragility is underestimated: Token-loop collapse bugs (same token repeating) traced to GPU cluster-specific race conditions in kernel interconnect speeds — meaning identical weights on different hardware produce different reliability profiles. Operators should validate models per cluster, not just per checkpoint.
Quantization is not monotonically lossy: The KL-divergence-guided layer selection paper shows 20% higher throughput with better fidelity than naive PTQ — operators accepting vendor-default quantization may be leaving both speed and quality on the table. Kimi's vendor benchmark program (Kimi Vendor Verifier) is a model for how labs can hold inference providers accountable.
The training/inference boundary is dissolving for frontier operators: RL training pipelines are bottlenecked by inference rollout speed (vLLM throughput), and production inference traces are the highest-signal post-training data. Operators building agentic systems should architect for closed-loop trace collection and continuous fine-tuning from day one, not as a future add-on.

Full analysis

Two Baseten engineers, Philip Kiely and Ali Taha, spent two hours walking through what it actually takes to serve open-weight LLMs in production, and the headline for anyone running inference is simple: you are probably leaving a 4-6x speedup on the floor. Same weights, different provider, and the spread on OpenRouter runs 4-6x between the worst and best. That gap is your money.

Reversibility: Type 2, mostly. Swapping inference providers or turning on speculative decoding for one workload is cheap to undo. The Type 1 buried inside is architectural: whether you build for closed-loop trace collection and continuous fine-tuning from day one, because retrofitting that later is a rewrite.

What's actually being decided: not "which model," but "do we treat inference as a solved commodity we rent, or as an engineering surface we own." The episode argues the latter still has enormous gains left.

Forcing function: none hard. But NVFP4 on Blackwell, GLM-5.2's sparse attention, and Rubin's design choices mean the stack you tuned last year is already stale.

The Skeptic. Most of the 10x is real but it is not free, and it is not yours unless you run dedicated. Speculative decoding at 2x requires a draft model trained on your traffic. Prefill-decode disaggregation needs separate GPU pools. The KL-divergence quantization trick came from a 45-page intern paper, not a config flag. Every one of these gains is an engineering team-quarter, not a checkbox. And the token-loop collapse bug, where identical weights on different clusters produce different reliability, means you cannot even trust a checkpoint that worked last week on new hardware. To the PM: the cheap magic numbers everyone quotes assume a team that does not exist at most companies.

The Researcher. The genuinely new claim is that quantization is not monotonically lossy. Baseten's Joshua showed that picking which layers to quantize by predicted error direction, measured via KL divergence (how far the compressed model's output probabilities drift from the full model's), yields a model 20% more compressed and more faithful, because errors cancel across layers. That contradicts the naive "less precision equals less quality" intuition everyone ships on. Kiely's framing is right: when the field brags about 1% gains, inference is solved. Right now people publish 200%. To the PM: we are still in the era where careful engineering buys you double, not decimal points.

The Open-Source Advocate. This is the open-weight operator's playbook, and it is the whole point. The tricks only pay off because you hold the weights: custom speculators, custom quantization, Haley grafting Kimi's vision encoder onto GLM-5.2 by freezing both and training a few-million-parameter projector to 56% on MMLU Pro. You cannot do any of that on a closed API. But the gap is real too. Open-source video (Wan 2.2, 20B) is nowhere near Veo or Kling, because full attention over 35,000 tokens for five seconds of 480p is O(n²) and intractable. To the PM: on text and agents, open weights plus engineering closes the gap. On video, it does not, yet.

The Compute Pragmatist. The economics are blunt. At millions of tokens an hour, hourly dedicated billing beats per-token API, and only dedicated unlocks the 2x speculator and the custom quant. So the crossover is a volume question you can compute today. But Ali Taha names the real ceiling, and it is not GPU design. Faster NICs (the network chips moving data between servers) would give nearly 100x on disaggregated serving across nodes. Inter-node KV cache transfer is the blocker. Meanwhile Rubin is built to kill mega-kernels, so betting a team on CUDA kernel tuning is betting against NVIDIA's roadmap. To the PM: your leverage is moving to orchestration, not thread-level control.

The Builder. What ships Tuesday: measure your current provider spread before you build anything. If you are on a shared endpoint at high volume, a dedicated deployment with a traffic-trained speculator is the fastest real win, and it needs a rollback plan because the collapse bugs are hardware-specific. The bigger architectural call is Kiely's continual-learning loop: inference, learn from inference, post-train, A/B, better model. He puts it at a few months to a couple years. If you are building agents, wire trace collection in now, because the traces are your highest-signal training data and you cannot recover them retroactively.

Where the council splits:

The Researcher and the Skeptic disagree on whether the quantization result is a tool or a trophy. A 45-page paper proving errors cancel is beautiful; a team that can reproduce it per-model, per-cluster is rare. The gain is real and mostly unclaimable without serious staff.

The Open-Source Advocate and the Compute Pragmatist part on where the ceiling sits. One sees weight access as the unlock; the other says the unlock is NIC bandwidth nobody at your company controls. Both are right, which means your gains are capped by hardware you rent, not code you write.

The Builder and the Skeptic split on the continual-learning loop. Ship the trace pipeline now versus wait until the pattern stabilizes. Building it early is cheap insurance; building the full loop early is chasing a moving target.

What it hinges on: whether your token volume clears the dedicated-deployment crossover. Below it, this whole episode is interesting reading and you rent from whoever is fastest this month. Above it, owning the inference stack buys 4-6x, and the traces buy you a better model. The verify step is dull and mandatory: benchmark your live provider against Artificial Analysis, measure your actual tokens-per-hour, and load-test candidate models per cluster before trusting a checkpoint.

The one belief to interrogate before committing a team: do you believe the continual-learning loop lands in months, as Kiely says, or years. Architect the trace collection either way. Delay the closed loop until the tooling settles.

Prediction: By the end of Q2 2027, no major open-weight video model (the Wan line or a successor at comparable open scale) will match Veo or Kling on standard quality benchmarks for clips longer than 10 seconds.

Confidence: Medium. The O(n²) attention wall is a hard compute constraint, not a tuning gap.

Why: The episode pins open-source video's deficit on a specific mechanism: full attention over 35,000 tokens for five seconds of 480p at 16fps, which grows with the square of length and becomes intractable past short clips. Ali Taha's own escape hatches are either autoregressive video, which he calls "terrible quality" today, or an "insane leap in compute" for full attention over millions of tokens. Neither is a quarters-away fix, and long-form stitching of 7-second chunks drifts to black screen. The opposite outcome would require an architectural breakthrough in autoregressive video quality that no open lab is close to shipping, so the gap holding is the safer call.

Revisit by 2027-06-30: We're right if the top open-weight video model still trails Veo/Kling on human-preference or standard video benchmarks for 10-second-plus clips. We're wrong if an open model at Wan-comparable scale matches or beats them on long-form quality.

Listen to the episode →