Refacto AI

Podcast episode

How to Build an AI-Native Company Today

agents evals inference model-pricing open-weights

Nathaniel Whittemore and Alex Lieberman work through Lieberman's 30-feature framework for what it means to build an "AI-native" company. No lab announcements, no benchmarks. Pure playbook: map your processes, build one queryable data source, route cheap tasks to cheap models, treat context documents like code, and hand software agents (programs that act autonomously on your behalf) autonomy one rung at a time. Commenter Binti Jamil flagged the one governance move that costs nothing: assign a named human owner to every workflow, so when an agent breaks something, someone is accountable.

Three features are worth doing this quarter. Stand up a test harness that scores each new model against your actual tasks. Route cheap work to cheap models to cut your API bill. Name an owner per workflow. The other 27 features, including "let agents find their own path," are real risks until you have hard guardrails at the data layer.

The 30-item list is how consultants bill hours. Start with two things: the test harness and the routing layer. Both are cheap and reversible, and they force the discipline the rest of the list assumes you already have.

Full analysis

Your draft

Your draft

Nathaniel Whittemore spent an episode walking through Alex Lieberman's 30-feature list for what makes a company "AI-native." No model releases, no benchmarks, no lab news. It's a playbook: map your processes, build one queryable data source, route cheap tasks to cheap models, treat context documents like code, run standing tests on every new model, and hand agents autonomy one rung at a time. The reader question is simple. Which of these 30 items is real work you should start this quarter, and which is a slide that dies on contact with your actual org?

This is an easy-to-undo situation for the reader. Nothing here is a vendor contract or an architecture you can't back out of. You're deciding where to point attention, not signing anything. That means less deliberation, faster action, and a low tolerance for the items that are just vocabulary.

The Skeptic. Thirty features is a symptom, not a framework. Whittemore says the agentic transition "actually began" in 2026, but a list this long is how consultants bill hours, and half these items are the same idea wearing different numbers. "Context as code," "intelligence layer," "citizen developer SDLC" are 2019's data-catalog and low-code pitches with agent paint. The one concrete data point, Anthropic's claim that 60% of internal building starts from a shared-space Claude trigger, is Anthropic describing Anthropic. A company staffed entirely by people who build models is the worst possible sample for whether your finance team will do the same. Treat it as an existence proof, not a target.

The Builder. Strip the list and three things are worth doing Tuesday. First, a standing test harness: every time a new model drops, run it against your own tasks with a pass/fail score, so you're not guessing whether GPT-5.x or the next Claude is better for your workflow. Second, model routing, send cheap tasks to cheap models, expensive planning to expensive ones. That's a real bill reduction you control. Third, a named human owner per workflow, which commenter Binti Jamil flagged and which costs nothing but a spreadsheet. The rest, "give agents goals and guardrails and let them find their own path," sounds great and breaks the first time an agent finds a path through your production database.

The Open-Source Advocate. The most actionable claim is feature 21: fine-tune an open-weight model on your own data using reinforcement learning, a training method where the model gets rewarded for good outputs, for high-volume repetitive tasks. Whittemore is right that near-frontier open models make this viable, and he's right to hedge that it's only for orgs with existing technical muscle. The vendor-lock-in point has teeth. He cites Cursor's acquisition limiting model access as the reason to build your own agent scaffolding on an open foundation like DeepSeek's released harness. That's a genuine risk a buyer should price: the tool you standardize on today can get bought and start restricting which models it'll run.

The Compute Pragmatist. The through-line across five of the 30 features is cost-per-successful-task, and that's the item that actually moves a budget. Routing, tiered models, loading only the slice of context an agent needs, fine-tuning small open models instead of paying frontier per-token rates. This is inference economics graduating from an engineering footnote to a line the CFO reads. But note the tension with the fine-tuning pitch: rolling your own model means reserving GPU capacity, standing up a training pipeline, and owning the maintenance. For most mid-size orgs, routing to cheaper hosted models gets you 80% of the savings without a GPU cluster. The self-host story is real for the top decile and a distraction for everyone else.

Where the thoughtful people split: The Open-Source Advocate and Compute Pragmatist both love feature 21, and they disagree on who it's for. Fine-tuning an open model on first-party data is a genuine edge if you already have ML engineers and reserved compute. It's a money pit if you're a 40-person company that read a framework. The second fault line is Whittemore's "let agents find their own path" versus the Builder's insistence on hard guardrails. Autonomy without enforcement at the data layer isn't ambition, it's an incident waiting for a name.

What this actually hinges on: almost none of the hard problems in this list are things a frontier lab will solve for you. Context management, tests against your own processes, permissions at the data layer, and a named owner when an agent breaks something. Anthropic and OpenAI ship the models. The governance, the eval harness, and the accountability are yours to build, and that's where systems integrators and internal platform teams will spend the next 18 months. The reader's move is to ignore 25 of the 30 features and stand up two things this quarter: a test harness against your own tasks, and model routing. Both are cheap, both are reversible, both cut real cost. Prediction: By the end of 2027's Q1 upfront and budget-planning cycle, the "agent swarms generating hundreds-to-thousands of paid-ad creative variations as normal practice" pattern will still be confined to performance-marketing shops and app-install advertisers; it will not have become standard practice among brand advertisers at major holding companies.

Confidence: Medium. Compute is cheap. Brand approval loops are not.

Why: The bottleneck was never model cost. The same framework that predicts the creative swarm also spends most of its weight on governance, permissions, and named human owners, because that is where campaigns actually stall. App-install and direct-response advertisers already run mass variation because their approval loop is a performance number, not a human taste call. For brand advertisers, the constraint is legal, brand, and platform-policy review, and falling inference prices do not shorten that cycle. Nothing in the current market signals that major brand advertisers are ready to hand creative approval to an agent loop; the barbell pattern he describes is already the reality, and it is not moving toward the middle by early 2027.

Revisit by 2027-03-31: We're right if paid-media creative generation at scale remains concentrated in performance and app-install advertising and brand advertisers still gate creative through human review at the close of Q1 2027 budget cycles. We're wrong if a major brand advertiser or top-four holding company publicly runs agent-generated creative swarms as standard practice across brand campaigns before that date.

Comments