Podcast episode

Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

by Ken Rona ·July 10, 2026 ·Latent Space

ai-in-adtech build-vs-buy cloud-costs cost-compression engineering

TL;DR

Modal CTO Akshat Bubna joins Latent Space to discuss the company's $355M Series C and the architectural shift from developer experience (DX) to agent experience (AX). The episode is a deep technical dive into AI infrastructure: why Kubernetes fails for agentic/bursty workloads, how Modal's "supercloud" across 17 providers serves frontier AI companies, and the inference optimization stack (DeFlash speculative decoding, GPU snapshotting, Auto Endpoints) that Modal is building to compete with raw GPU rental. High relevance for AI infrastructure operators and applied-AI practitioners.

What was covered

Origin and positioning: Modal began as a serverless container runtime to fix Kubernetes's poor handling of bursty, compute-heavy workloads. Added GPU support a year before ChatGPT—before recognizing how large the market would become. Now explicitly positioned as a cloud platform built from scratch for AI workloads: inference, training, batch, and sandboxes.
Developer Experience → Agent Experience (AX): Modal has retooled its SDK team around agent experience. The key claim: agents benefit from the same decorator-based, self-provisioning infrastructure as developers—less YAML, more programmatic control. Observability (dashboards, CLI-accessible logs) is now more important than code readability since agents write but don't explain code.
Elastic inference and GPU snapshotting: Modal's largest use case is elastic inference for custom models (Suno/audio, Runway/video, robotics, computational bio). GPU snapshotting—capturing GPU state including compiled model weights—dramatically reduces cold-start latency and enables true scale-to-zero with fast burst recovery.
DeFlash and speculative decoding: Modal open-sourced DeFlash, a block-based speculative decoder (a technique using a smaller "draft" model to predict tokens in batches that a larger model then verifies—achieving 2–4× throughput gains with no quality loss). Modal is upstreaming improvements into SGLang. Auto Endpoints wraps these optimizations into a no-code deployment path, with planned automatic draft-model retraining as user data distributions shift.
RL rollouts and 100,000 sandboxes: Reinforcement learning (RL) training rollouts are described as the most extreme burstiness case—requiring up to 100,000 simultaneous sandboxes. Sandboxes now support Docker Compose-style sidecars, private IPv6 overlay networking (I6PN), mTLS tunnels, and RDMA at 3 Tbps for multi-node distributed training.
17-cloud supercloud strategy: Modal does not own data centers—it abstracts capacity across 17 cloud and neo-cloud providers, with a proprietary reliability layer that masks individual provider failures. This enables regional pinning (EU, AU, US), fungible GPU/CPU capacity, and a "compute strategy" team that hedges reservations (1-year vs. 3-year) across GPU types and regions.
Gitpod/Ona → OpenAI Codex Cloud: Akshat notes Gitpod (rebranded Ona) was technically strong but targeted build-time sandboxes while the market moved to runtime sandboxes. The team's acquisition by OpenAI is expected to produce "Codex Cloud"—viewed as a serious competitive entrant.
Modal Bench: A benchmark Modal is building to identify where current LLMs (described as capable post-Claude 4, poor before) fail on Modal-specific tasks (e.g., reading logs and diagnosing failures), feeding directly into product/CLI surface improvements.

Notable claims & predictions

Akshat Bubna: "Pre-Claude 4 [LLMs] were not [good at generating Modal code], and then now they're able to one-shot stuff out of the box." — Direct capability milestone tied to a specific model generation.
Akshat Bubna: "When you're doing RL rollouts, you sometimes need 100,000 sandboxes." — Quantifies the infrastructure scale implied by modern RLHF (Reinforcement Learning from Human Feedback) / RL training pipelines.
Akshat Bubna on LLM-mediated permissions: "I'm skeptical of LLM-mediated permissions for stuff that is at the sandbox level because you do want hard boundaries. Otherwise, someone can exfiltrate stuff." — A direct pushback on the "LLM as kernel" framing for agentic systems.
Akshat Bubna on managed agents (OpenAI/Anthropic/Google): "The managed agent thing is a great place to start if you're starting out building an agent, but when you get to production grade... you need a lot more control over your compute primitive." — Positions frontier lab agent offerings as top-of-funnel, not direct competition.
Akshat Bubna on speculative decoding: "Improving a kernel will only give you a few percentage points of improvement, and increasing accept length [in speculative decoding] is literally a multiplicative decrease—like 2–4×." — Argues accept-length optimization is the dominant inference performance lever, not kernel tuning.
Swyx (host), paraphrasing Ethan from xAI/Groq: "The next tier in video gen is not a better video model, it's a better model or agent that orchestrates video models." — Prediction that agentic video pipelines will matter more than raw generative model quality.

Why this matters for AI operators

Inference economics are shifting from raw GPU rental toward managed optimization stacks. Modal's Auto Endpoints + DeFlash + speculative decoding represent a productized path to 2–4× throughput gains on open models—directly affecting cost-per-token for operators running custom or fine-tuned models. The open-sourcing of DeFlash raises the floor for what any operator can achieve without Modal.
RL training infrastructure requirements are now quantifiable and extreme. The 100,000-sandbox figure for RL rollouts is a concrete planning number for any lab or applied-AI team considering on-premises vs. cloud RL pipelines. Serverless multi-node training (RDMA, 3 Tbps) for post-training runs is becoming a real product category, not just a hyperscaler feature.
The "agent cloud" stack is consolidating around sandbox + persistent storage + network control, not just LLM APIs. Frontier companies (Cognition, Ramp) are converging on Modal-style infrastructure for production agents. Operators building agent pipelines should evaluate whether managed agent offerings from OpenAI/Anthropic/Google provide sufficient compute control, or whether a specialized sandbox layer is required.
GPU-to-CPU ratio normalization has infrastructure implications. The shift from ~8:1 GPU/CPU toward ~1:1 in agentic workloads (per Jensen Huang's GTC framing, echoed here) means operators need co-located, elastically scalable CPU capacity—not just GPU burst. Multi-cloud providers with geographic flexibility (Modal's 17-cloud model) become more relevant as low-latency CPU proximity to GPU sandboxes matters for agent responsiveness.

Full analysis

Modal's $355M Series C funds a bet that AI infrastructure is splitting into a new category: not raw GPU rental, not managed agent APIs, but a "supercloud" layer that abstracts compute across 17 providers and optimizes inference for you. The pitch to anyone building AI features — including ad-tech teams shipping bidding agents, creative generators, or brand-safety classifiers — is that you can get 2–4× more throughput per GPU dollar without owning the optimization stack yourself.

Reversibility: Mostly Type 2 for the reader. Trying Modal (or a rival serverless GPU platform) for one inference workload is a reversible experiment. The Type 1 decision hiding underneath is architectural: do you build agent pipelines around a specialized sandbox/compute layer, or lean on managed agents from OpenAI/Anthropic/Google? That choice compounds.

What's actually being decided: Not "should I use Modal." It's "where does my agent's compute live, and who owns my inference cost curve?" For ad-tech specifically: as real-time bidding and creative generation get LLM-mediated, does the decisioning logic run inside a managed API you can't tune, or on infrastructure where accept-length and cold-start are yours to optimize?

Forcing function: Surging inference costs (the reader's own saved reading flags this today), plus the "post-Claude 4 models can one-shot infra code" milestone. No hard deadline, but the cost pressure is real now.

The Skeptic — The 2–4× throughput number comes from speculative decoding, which is real and well-understood — but it's a technique, not a moat. Modal open-sourced DeFlash and is upstreaming into SGLang, which means the floor rises for everyone and Modal's own edge erodes. The "17-cloud reliability layer" is the actual product, and it's unfalsifiable in a podcast: masking provider failures sounds great until a correlated outage across three neo-clouds takes your RL run down at once. And 100,000 sandboxes for RL rollouts is a frontier-lab number — the median ad-tech team runs zero RL and won't for years. For a PM: most of this is built for the handful of companies training their own models, not the hundred using someone else's API.

The Researcher — The genuinely interesting claim is empirical: LLMs went from useless to one-shotting Modal-specific code across the Claude 4 boundary. That's a real capability step, and Modal Bench (testing whether models can read logs and diagnose failures) is a smart eval because it targets the boring operational tasks agents actually fail at, not toy coding puzzles. Speculative decoding math checks out — accept-length gains are multiplicative (2–4×) while kernel tuning is additive (single-digit %), so the emphasis is correct. For a PM: the smartest bet here isn't the infrastructure, it's the observation that agents now write code well but can't debug their own failures — and whoever fixes debugging wins.

The Open-Source Advocate — This is the good-news story. DeFlash is open-sourced and improvements go upstream into SGLang, so an ad-tech team running a fine-tuned model on rented H100s can capture most of the throughput gain without paying Modal a markup. The barrier to a 2–4× cost cut isn't proprietary anymore — it's engineering time. That reframes the buy-vs-build: Modal is selling you the time and the reliability layer, not the technique. For a PM: the core cost-saving trick is now free; you're paying Modal for convenience and uptime, not secret sauce — so price it against your own engineers' hours.

The Compute Pragmatist — The load-bearing insight for ad-tech: the GPU-to-CPU ratio is normalizing from ~8:1 toward ~1:1 in agentic workloads. Real-time bidding is CPU-latency-sensitive and geographically pinned — an auction has to clear in ~100ms in a specific region. If your agentic decisioning needs GPU inference co-located with low-latency CPU near the ad exchange, a multi-region abstraction layer genuinely matters. GPU snapshotting enabling true scale-to-zero is the real cost lever for bursty, spiky ad traffic — you don't pay for idle GPUs between bid-request storms. For a PM: ad auctions are bursty and location-locked, which is exactly the workload this is built for — more so than for a chatbot.

The Enterprise Buyer — Here's where the ad-tech reader should slow down. Modal doesn't own data centers; it hedges reservations across 17 third parties. For a media buyer moving client budgets or a publisher handling first-party data, that's a data-residency and auditability question, not a throughput question. Regional pinning (EU, AU, US) helps, but "our reliability layer masks provider failures" is not the same as an SLA you can put in a contract or an indemnification clause. For a PM: your procurement and privacy teams will care more about which of the 17 clouds your data touches than about tokens-per-second.

Where the council splits:

Moat vs. commodity. The Open-Source Advocate says Modal just gave away its best trick, so the technique is commoditized — you can DIY the 2–4×. The Compute Pragmatist says the technique was never the product; the multi-region reliability-and-snapshotting layer is, and that's hard to replicate. Both are right, which means Modal's value is real but narrower than the throughput headline suggests.
Who this is even for. The Skeptic says 100,000-sandbox RL is frontier-lab theater irrelevant to ad-tech. The Compute Pragmatist counters that scale-to-zero on bursty, region-pinned GPU is exactly the RTB workload profile. The resolution: ignore the RL numbers, watch the burst-inference story.
Build vs. buy the agent layer. The Researcher and Builder see managed agents (OpenAI/Anthropic/Google) as fine for prototypes; Bubna's own "top-of-funnel, not competition" framing admits production needs more compute control. The Enterprise Buyer notes that for regulated ad/data workflows, "more control" cuts both ways — more control also means more of your own compliance burden.

What it hinges on: For the ad-tech reader, the decision turns on one belief — is your AI workload bursty and latency-pinned enough that a specialized serverless GPU layer beats both raw rental and managed APIs? If you're running real-time agentic bidding or spiky creative generation, plausibly yes. If you're calling an LLM API a few thousand times a day for content tagging, no — this is over-engineering.

What to verify before committing: Run your actual traffic pattern against scale-to-zero cold-start latency (the snapshotting claim is testable in an afternoon). Price the DeFlash-on-SGLang DIY path against Modal's managed markup using your engineers' real hourly cost. And get your privacy team to enumerate which of the 17 underlying clouds would touch first-party or client data before anything production-grade.

Direct ad-tech impact: moderate, and indirect. This episode is not about advertising. But the cost curve it describes — inference optimization moving from raw rental to managed 2–4× throughput stacks — is the same curve that determines whether LLM-mediated bidding, creative gen, and brand-safety classification are economically viable at RTB scale. The containerization-of-the-supply-chain theme in the reader's own AdExchanger saved reading (custom AI models moving into SSP environments) rhymes with this exactly: decisioning is moving to wherever the flexible compute lives.

Prediction: By NeurIPS 2026 (early December), block-based speculative decoding of the kind Modal open-sourced as DeFlash will be a standard, upstreamed feature in the major open inference servers (SGLang and vLLM), available to any operator without a Modal contract.

Confidence: High — Modal is already upstreaming into SGLang, and vLLM ships speculative decoding today.

Why: Speculative decoding gives multiplicative throughput gains with no quality loss, so every open inference server has strong incentive to adopt the best variants, and Modal is actively contributing the code upstream rather than keeping it proprietary.

Revisit by 2026-12-10: We're right if block/batch-based speculative decoding with automatic draft-model handling is documented and usable in mainline SGLang or vLLM releases. We're wrong if the technique remains gated behind Modal's proprietary Auto Endpoints with no open equivalent shipped.

That commoditization is the real ad-tech takeaway: the throughput savings will be available to anyone, so the durable question for ad-tech operators isn't whether to get 2–4× cheaper inference — it's whether their bursty, region-locked workloads justify paying someone else to run the reliability layer.

Listen to the episode →