From Voice Agents to AI Avatars with Alexander Smola - #777

Full analysis

Alexander Smola, co-founder of Boson AI and a Carnegie Mellon professor, went on TWIML to argue that the winning voice-AI product is the one you can afford to run. Biggest model loses if nobody can pay to run it.

What's being weighed here isn't a decision you make. It's whether to believe a specific claim about voice AI, and whether to change how you buy it. That's easy to undo: nobody's signing a contract off a podcast. Nothing sets a deadline. So the question is narrow. Is "price-first" voice AI real and durable, or is this a founder talking his book before a fundraise?

The Skeptic

Smola compares against GPT and Google with the model's reasoning step switched off. That's the catch. He turned off the exact feature that makes these models good at hard, multi-step requests, then declared victory at a tenth the cost. Of course it's cheaper. It's doing less. And the benchmarks are BigBench Audio and Complex Function Bench, not names most buyers can independently check. Every voice demo looks great until a real customer talks over the agent in a loud room with an accent the training data thinned out. The one live anecdote, Slovenian to Hindi in a bar, was helped by the iPhone's own noise cancellation. iOS did the work here.

The Researcher

Strip the marketing and the interesting part is the architecture. Boson splits the job in two: a light, fast model that talks, and a heavier model behind it that reasons and calls tools. That's a real answer to a real constraint. Audio needs more than ten tokens a second versus three to five for text, so a fast, affordable voice model has to be small. The 150-millisecond interruption target matches how fast a human brain reacts, and their ProActBench and IH Bench measure whether an agent knows the difference between someone interrupting and someone just saying "uh-huh." Existing text benchmarks miss that entirely. The 100 million hours of audio is the actual barrier. You can't fine-tune your way to that.

The Open-Source Advocate

One Boson audio release was built on Meta's Llama, which matters more than the cost claim. It says you can take open weights and add a voice layer without starting from scratch, as long as you do the full pre-, mid-, and post-training so the model doesn't forget how to reason when you bolt audio on. That's the recipe. But the recipe isn't the moat. The 100 million hours of audio and the pipelines that clean it are private, and no permissive license gets you those. So this is a familiar shape: open base model, closed data, closed product. The open ecosystem gets the starting point and none of the finish.

The Compute Pragmatist

Smola said the thing frontier labs won't: if a single conversation needs a whole Blackwell server GPU, your customers can't afford it, no matter how gorgeous the demo. That's the discipline. In voice, KV-cache is what kills the budget. That's the running memory of everything said so far in the conversation, and it grows every second the call continues. Long calls get expensive fast. Splitting into a small talker and an occasional reasoner is how you keep the per-minute cost survivable. Owning the data center matters too. He said storing 100 million hours on a rented cloud would "eat you alive." That's real. Egress and storage bills sink audio companies quietly.

The Builder

What would I ship Tuesday? Nothing from Boson yet, because I can't buy proven latency and multi-tool support off a podcast. But the design tells me what to demand from any voice vendor. Ask for per-conversation cost at your concurrency, not per-token. Ask what happens to cost on a 20-minute call. Ask them to run ProActBench and IH Bench in front of you, because a voice agent that treats "hai, hai" as an interruption will infuriate every Japanese user you have. And ask what happens when the reasoning step is on, since Smola's own numbers assume it's off. The gap between "fast and cheap" and "actually handles the tool call" is where your on-call engineer gets paged.

Where they disagree

The Skeptic and the Researcher split on the cost claim. The Skeptic says a tenth the cost with reasoning off is a rigged comparison. The Researcher says the two-stage split is a genuine way to get most of the quality for a fraction of the money, and turning off reasoning to hold real-time speed is a defensible product choice, not a cheat. Both can be right: the architecture is sound and the benchmark framing is generous.

The Open-Source Advocate and the Compute Pragmatist agree on the shape but weigh it differently. Open base model, closed data. The Advocate sees a ceiling for anyone without their own audio hoard. The Pragmatist sees why that hoard exists: the economics of storing and processing 100 million hours forced Boson to own infrastructure most competitors won't build.

What it hinges on

Two facts. First, does the cost advantage hold when the reasoning step is on and the agent has to actually complete tasks, not just chat fast? That's the number a buyer lives with. Second, does the audio-data moat translate into quality a customer can feel, or just a bigger storage bill? If you're evaluating voice vendors this year, don't take a benchmark slide. Design one 15-minute call in a noisy room, in two languages, with three tool calls, and price it at your real concurrency. That single test settles more than any leaderboard.

Prediction Prediction: Boson AI's headline claim of beating OpenAI and Google on audio benchmarks "at one-tenth the cost" will not be reproduced by any independent third-party audio-benchmark comparison with the reasoning step enabled by 2027-03-22; the advantage shown publicly will remain in the reasoning-disabled configuration Alex Smola used.

Confidence: Medium. Vendors never publish the harder comparison that narrows their own headline number.

Why: Smola's own framing is that the one-tenth-cost win comes with chain-of-thought (the slower step where a model works through a problem before answering) switched off to keep replies fast. That configuration inflates any cost-per-quality comparison, because turning off reasoning cuts compute regardless of whose model you run. Boson is selling real-time affordability, so it will keep publishing the number that flatters that pitch and has no reason to publish the reasoning-on version that narrows the gap. The opposite outcome requires someone with no stake to build audio-eval infrastructure that barely exists yet, and Boson to welcome a test that can only hurt the headline figure.

Revisit by 2027-03-22: We're right if no independent, reasoning-enabled comparison confirms a roughly 10x cost advantage on audio benchmarks by that date. We're wrong if a third party publishes one showing Boson at or near one-tenth the cost of GPT or Gemini with reasoning on.

The architecture is the real lesson regardless of how the number holds up. Cost-first design, split into a fast talker and a slower reasoner, is worth copying into your own vendor questions whether or not Boson turns out to be the one you buy.