Airbnb Is Building a ‘Company Brain’ for AI Agents with Ahmad Al-Dahle - #781

Full analysis

Ahmad Al-Dahle, the CTO of Airbnb and the man who used to run Meta's Llama models, spent an hour describing how one large consumer company actually wires AI into its guts. Strip away the branding and the news is this: a company with real scale is telling you, out loud, that frontier models are overkill for most of what it ships, that evals have gotten so saturated nobody can tell the models apart, and that AI handles half its support chats today with a hard ceiling it cannot climb.

This is a briefing, not a decision. Nothing here is hard to undo, because you are not Airbnb and you are not buying anything from them. What is being decided, for the reader, is where to spend your own AI budget over the next two quarters: on the biggest model, or on the plumbing around a smaller one. No deadline sets the clock. But the signal is strong enough to act on now.

The Skeptic

The 50% support-resolution number is the one everyone will quote, and it is softer than it sounds. Half of chats resolved by AI, sure, but Al-Dahle tells you which half: informational queries. "When does check-in start," "how do I change my dates." The AI never touches the cases that actually cost Airbnb money, the two-party disputes where one person uploads a faked photo of a trashed kitchen. Those stay with humans, and he is blunt that today's models "can't really do a great job of mediating." AI cleared the easy half of the queue. Every support team already knew the easy half was clearable. The hard half is where the ceiling is, and it hasn't moved.

The Researcher

The eval story is instructive. Airbnb runs its own benchmarks, scored by an LLM acting as judge and calibrated against a panel of senior engineers. Open and closed models both pile up at 92 to 93%. When everything scores the same, your benchmark has stopped measuring anything. Al-Dahle names the fork honestly: either the tests are too easy, or the models are genuinely good enough and the only remaining question is price. Either way, the capability race as a buying signal is dead for a huge class of everyday tasks. If a $3-per-million-token model and a $15 one both clear your bar, you are not choosing on quality anymore. You are choosing on cost. That is a different world than the one the launch-day benchmark charts are selling you.

The Open-Source Advocate

Here is a man who built Llama saying, from inside a Fortune-500 consumer company, that for search, which is his highest-traffic workload, he takes "the smallest model possible" and specializes it with his own training rather than calling a frontier model at all. He names NVIDIA's Nemotron, a small open-weight model, as one in the rotation. He credits OpenAI specifically for open-sourcing its Codex coding tool and folds it into Airbnb's internal setup. The pattern is: big closed models for broad reasoning, small open ones that you fine-tune for the narrow, high-volume jobs. That split is now the default architecture at serious shops, and it quietly inverts the "just call GPT for everything" advice that was gospel eighteen months ago.

The Compute Pragmatist

Follow the money and it points the same way. Search runs at enormous query volume, every millisecond and every fraction of a cent multiplied across millions of requests a day. At that scale a frontier model's price and latency are disqualifying no matter how smart it is. So Airbnb distills, which means compressing a big model's know-how into a small one you can run cheap and fast, and post-trains it on the one narrow task that matters. The frontier labs still win the reasoning layer and the coding harness. But the volume layer, the part that actually sets your inference bill, is being colonized by small specialized models and the training infrastructure to shape them. If you are buying AI by the token for high-traffic work, your spending map should look more like Airbnb's than like a single frontier contract.

The Builder

Three things here you can use on Tuesday. First, Project Everest, their "company brain," and the warning that comes with it: dump stale notes into a markdown pile and a confident model will serve the wrong answer as fact. He calls it context poisoning. If you are building an internal knowledge assistant, the content rots faster than you think, and the model never flags it. Second, progressive disclosure: feed the agent just enough to take one step, then let it ask for more, instead of pouring the whole folder in up front. That is a concrete fix for flaky agents, available today. Third, he rejects token counts as a productivity metric outright. Low usage might mean someone is stuck; high usage might mean waste. He investigates outliers instead of capping them. If your org is running a token leaderboard, you are measuring the wrong thing.

The tensions

The real disagreements are two. The Researcher says the models are already good enough and the game is now cost. The Builder says good enough at the model layer just moves the hard work into the plumbing: the knowledge store that poisons itself, the agent that gets too much context, the eval that can't tell anything apart. Both are right, and that is the point. The capability ceiling stopped being your problem; the engineering around it became your whole problem.

The second tension is the Skeptic against everyone's enthusiasm for that 50% figure. The Open-Source Advocate and the Compute Pragmatist see a mature, cost-aware architecture and want to copy it. The Skeptic notes that the number tops out exactly where the money is, in adversarial disputes, and no amount of clever routing fixes a model that can't referee two people lying to it.

What it hinges on

For the reader, the decision rests on one belief: is the quality race over for the work you actually ship? Airbnb's eval clustering says yes for the ordinary stuff and a flat no for anything adversarial or high-trust. So the move is to run your own eval, calibrated against your own experts, and watch whether your candidate models cluster the way Airbnb's did. If they do, stop paying frontier prices for that task and distill down. If they spread out, you still have a real quality choice to make. Either way, the days of picking a model off a public leaderboard and calling it done are gone.

Prediction: Airbnb's publicly reported AI customer-support resolution rate will still be at or below 60% at its next disclosure tied to a 2026 full-year or Q4 earnings update in early 2027.

Confidence: Medium. The ceiling is the unsolved mediation problem Al-Dahle names directly in the source.

Why: Al-Dahle says plainly that the resolved half is informational queries and that the unresolved cases are adversarial two-party disputes where evidence can't be verified, and that "the models today can't really do a great job of mediating." That ceiling is a model-capability limit, not a staffing or tooling gap Airbnb can engineer around in a few months. The cases left in the queue are the hard ones by definition, so each further point of resolution gets more expensive, which is why a jump to 70%-plus would require a genuine leap in multi-party reasoning that no current model has shown. The opposite outcome, a fast climb past 60%, would mean frontier models suddenly got good at refereeing disputes between two people with incentives to lie, and nothing in the last year of releases suggests that.

Revisit by 2027-04-26: We're right if Airbnb's next reported full-AI support resolution figure is 60% or lower. We're wrong if it is above 60%.