Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

Full analysis

Modal's $355M Series C funds a bet that AI infrastructure is splitting into a new category: not raw GPU rental, not managed agent APIs, but a "supercloud" layer that abstracts compute across 17 providers and optimizes inference for you. The pitch to anyone building AI features — including ad-tech teams shipping bidding agents, creative generators, or brand-safety classifiers — is that you can get 2–4× more throughput per GPU dollar without owning the optimization stack yourself.

Reversibility: Mostly Type 2 for the reader. Trying Modal (or a rival serverless GPU platform) for one inference workload is a reversible experiment. The Type 1 decision hiding underneath is architectural: do you build agent pipelines around a specialized sandbox/compute layer, or lean on managed agents from OpenAI/Anthropic/Google? That choice compounds.

What's actually being decided: Not "should I use Modal." It's "where does my agent's compute live, and who owns my inference cost curve?" For ad-tech specifically: as real-time bidding and creative generation get LLM-mediated, does the decisioning logic run inside a managed API you can't tune, or on infrastructure where accept-length and cold-start are yours to optimize?

Forcing function: Surging inference costs (the reader's own saved reading flags this today), plus the "post-Claude 4 models can one-shot infra code" milestone. No hard deadline, but the cost pressure is real now.

The Skeptic — The 2–4× throughput number comes from speculative decoding, which is real and well-understood — but it's a technique, not a moat. Modal open-sourced DeFlash and is upstreaming into SGLang, which means the floor rises for everyone and Modal's own edge erodes. The "17-cloud reliability layer" is the actual product, and it's unfalsifiable in a podcast: masking provider failures sounds great until a correlated outage across three neo-clouds takes your RL run down at once. And 100,000 sandboxes for RL rollouts is a frontier-lab number — the median ad-tech team runs zero RL and won't for years. For a PM: most of this is built for the handful of companies training their own models, not the hundred using someone else's API.

The Researcher — The genuinely interesting claim is empirical: LLMs went from useless to one-shotting Modal-specific code across the Claude 4 boundary. That's a real capability step, and Modal Bench (testing whether models can read logs and diagnose failures) is a smart eval because it targets the boring operational tasks agents actually fail at, not toy coding puzzles. Speculative decoding math checks out — accept-length gains are multiplicative (2–4×) while kernel tuning is additive (single-digit %), so the emphasis is correct. For a PM: the smartest bet here isn't the infrastructure, it's the observation that agents now write code well but can't debug their own failures — and whoever fixes debugging wins.

The Open-Source Advocate — This is the good-news story. DeFlash is open-sourced and improvements go upstream into SGLang, so an ad-tech team running a fine-tuned model on rented H100s can capture most of the throughput gain without paying Modal a markup. The barrier to a 2–4× cost cut isn't proprietary anymore — it's engineering time. That reframes the buy-vs-build: Modal is selling you the time and the reliability layer, not the technique. For a PM: the core cost-saving trick is now free; you're paying Modal for convenience and uptime, not secret sauce — so price it against your own engineers' hours.

The Compute Pragmatist — The load-bearing insight for ad-tech: the GPU-to-CPU ratio is normalizing from ~8:1 toward ~1:1 in agentic workloads. Real-time bidding is CPU-latency-sensitive and geographically pinned — an auction has to clear in ~100ms in a specific region. If your agentic decisioning needs GPU inference co-located with low-latency CPU near the ad exchange, a multi-region abstraction layer genuinely matters. GPU snapshotting enabling true scale-to-zero is the real cost lever for bursty, spiky ad traffic — you don't pay for idle GPUs between bid-request storms. For a PM: ad auctions are bursty and location-locked, which is exactly the workload this is built for — more so than for a chatbot.

The Enterprise Buyer — Here's where the ad-tech reader should slow down. Modal doesn't own data centers; it hedges reservations across 17 third parties. For a media buyer moving client budgets or a publisher handling first-party data, that's a data-residency and auditability question, not a throughput question. Regional pinning (EU, AU, US) helps, but "our reliability layer masks provider failures" is not the same as an SLA you can put in a contract or an indemnification clause. For a PM: your procurement and privacy teams will care more about which of the 17 clouds your data touches than about tokens-per-second.

Where the council splits:

1. Moat vs. commodity. The Open-Source Advocate says Modal just gave away its best trick, so the technique is commoditized — you can DIY the 2–4×. The Compute Pragmatist says the technique was never the product; the multi-region reliability-and-snapshotting layer is, and that's hard to replicate. Both are right, which means Modal's value is real but narrower than the throughput headline suggests.

2. Who this is even for. The Skeptic says 100,000-sandbox RL is frontier-lab theater irrelevant to ad-tech. The Compute Pragmatist counters that scale-to-zero on bursty, region-pinned GPU is exactly the RTB workload profile. The resolution: ignore the RL numbers, watch the burst-inference story.

3. Build vs. buy the agent layer. The Researcher and Builder see managed agents (OpenAI/Anthropic/Google) as fine for prototypes; Bubna's own "top-of-funnel, not competition" framing admits production needs more compute control. The Enterprise Buyer notes that for regulated ad/data workflows, "more control" cuts both ways — more control also means more of your own compliance burden.

What it hinges on: For the ad-tech reader, the decision turns on one belief — is your AI workload bursty and latency-pinned enough that a specialized serverless GPU layer beats both raw rental and managed APIs? If you're running real-time agentic bidding or spiky creative generation, plausibly yes. If you're calling an LLM API a few thousand times a day for content tagging, no — this is over-engineering.

What to verify before committing: Run your actual traffic pattern against scale-to-zero cold-start latency (the snapshotting claim is testable in an afternoon). Price the DeFlash-on-SGLang DIY path against Modal's managed markup using your engineers' real hourly cost. And get your privacy team to enumerate which of the 17 underlying clouds would touch first-party or client data before anything production-grade.

Direct ad-tech impact: moderate, and indirect. This episode is not about advertising. But the cost curve it describes — inference optimization moving from raw rental to managed 2–4× throughput stacks — is the same curve that determines whether LLM-mediated bidding, creative gen, and brand-safety classification are economically viable at RTB scale. The containerization-of-the-supply-chain theme in the reader's own AdExchanger saved reading (custom AI models moving into SSP environments) rhymes with this exactly: decisioning is moving to wherever the flexible compute lives.

Prediction: By NeurIPS 2026 (early December), block-based speculative decoding of the kind Modal open-sourced as DeFlash will be a standard, upstreamed feature in the major open inference servers (SGLang and vLLM), available to any operator without a Modal contract.

Confidence: High — Modal is already upstreaming into SGLang, and vLLM ships speculative decoding today.

Why: Speculative decoding gives multiplicative throughput gains with no quality loss, so every open inference server has strong incentive to adopt the best variants, and Modal is actively contributing the code upstream rather than keeping it proprietary.

Revisit by 2026-12-10: We're right if block/batch-based speculative decoding with automatic draft-model handling is documented and usable in mainline SGLang or vLLM releases. We're wrong if the technique remains gated behind Modal's proprietary Auto Endpoints with no open equivalent shipped.

That commoditization is the real ad-tech takeaway: the throughput savings will be available to anyone, so the durable question for ad-tech operators isn't whether to get 2–4× cheaper inference — it's whether their bursty, region-locked workloads justify paying someone else to run the reliability layer.