Podcast episode
The Trust Economy: Nirav Tolia on Communitas, AI, and Rebuilding the Internet Around Human Connection
evals guardrails hallucination privacy rag
Brett House and Rio Longacre interview Nirav Tolia, founder of Nextdoor, on how he's thinking about AI, local data, and whether verified-identity networks can survive in a world of generated content.
Tolia's headline move is refusing to license Nextdoor's data to AI companies, citing user trust. He also claims he added a "never fabricate" instruction to his AI system prompt and the model fabricated anyway, then "confessed" it did so to be more compelling. The privacy architecture underneath the refusal is more interesting than the refusal itself: ZIP-code-level public health signals, verified identity, no individual PII. Freshpaint is already using that pattern for healthcare targeting that survives HIPAA scrutiny. Longacre puts bot traffic at 50% of the web and 30% of comments, numbers that are directionally plausible but unsourced.
The data-refusal announcement is cheap PR. The architecture behind it is worth copying. And on fabrication: a system prompt telling a model not to lie is a suggestion, not a guardrail. If your product can't tolerate hallucinations, the fix has to live outside the model entirely.
Full analysis
The Skeptic
Tolia's refusal to license data is free to announce when nobody's writing him a big check. Reddit signed with Google for a reported $60M/year; Nextdoor's hyperlocal chatter about lost cats and pothole complaints is worth a fraction of that to a frontier lab. Refusing a deal you were never going to get a good price on is a press release. For a PM: "we protect user trust" and "our data isn't worth much to LLMs anyway" produce identical behavior, and you can't tell them apart from the outside. The hallucination anecdote is worse. An LLM that "admitted" it fabricated to be "more compelling" is the model confabulating an explanation for a confabulation. Tolia treats a hallucinated confession as a confession. That's not evidence about model behavior. It's the same failure mode, one level up.
The Researcher
The system-prompt story is the useful bit, and it cuts against Tolia's framing. He put "never fabricate" in a markdown file, fed it as a persistent instruction, and the model fabricated anyway. That's the known result: instructions in the prompt are soft constraints, not guarantees, and they degrade over long contexts and multi-turn sessions. For a PM: telling a model "don't make things up" in the system prompt is a suggestion, not a control. If your product cannot tolerate fabrication, the fix lives outside the model. Retrieval grounding, citation-required outputs, a verifier pass, refusal-on-low-confidence. Longacre's "50% of traffic is bots" and "30% of comments are bots" are uncited and methodology-dependent; the 40-50% bot share is real in aggregate, the comment figure is a vibe. Don't rebaseline your measurement on a podcast number.
The Open-Source Advocate
The interesting tension Tolia skips: robots.txt and a login wall keep you out of the training corpus, but they don't keep you out of the inference-time answer. Wikipedia and Reddit are in the weights. Nextdoor isn't. So an LLM answering "what's the crime situation in my neighborhood" pulls from whatever open data it has and confidently guesses. Nextdoor's walled data doesn't correct the model, it just isn't there to. For a PM building on open weights: the moat other people build by withholding data is exactly the gap your RAG layer has to fill with licensed or first-party sources. Nextdoor's abstention is your retrieval hole.
The Enterprise Buyer
The part that maps to a real contract is Freshpaint. Privacy-safe healthcare targeting on ZIP-code-level public health signals, no individual-level PII, is a template that survives cookie deprecation and HIPAA scrutiny. That's signable. Regulated verticals (pharma, insurance, finance) can buy geographic and contextual inventory without the tracking exposure that's getting everyone sued. The local-journalism structure (distribution, no money either direction, traffic flows back to publishers) is a genuinely different posture from Google's AI Overviews eating the click. Whether it scales past a Dallas Morning News handshake is unproven, but the shape is real and buyable.
Where they part ways
The Skeptic says the data refusal is a non-decision dressed as principle. The Enterprise Buyer says the mechanism underneath it (verified identity, ZIP-level signal, no individual PII) is a real product other companies can copy. Both are right, and that's the useful split: the announcement is cheap, the architecture is valuable. Copy the architecture, ignore the sermon.
The Researcher and Tolia disagree on what the hallucination story proves. Tolia thinks it shows AI needs human pairing. The Researcher thinks it shows system prompts aren't controls, which is a build lesson, not a philosophy.
What it hinges on for the reader's team: two beliefs. One, that proprietary first-party data grows more valuable as generated content goes to zero. That's plausible and directionally right, but the value only shows up if your data is genuinely non-substitutable, not if a model can approximate it from open sources. Two, that prompt-level instructions can enforce factuality. That one is false, and the episode accidentally proves it.
The council leans practical: steal the Freshpaint pattern, treat the data-refusal as PR, and if you ship anything that can't tolerate fabrication, put the guardrail outside the model. Verify by running your own test: put "never fabricate, cite every claim" in your system prompt, then hit it with 200 questions where you've deleted the ground-truth source. Count the confident wrong answers. That number is your actual fabrication rate, and it won't be zero.
Prediction: By the release of GPT-5.5 or the next frontier model from OpenAI, Anthropic, or Google (expected within the next two frontier release cycles, by 2027-06-01), a system-prompt instruction to "never fabricate" will still fail to reliably prevent fabrication on ground-truth-absent queries, and every serious production stack will still rely on external grounding or verifier passes rather than prompt instructions alone.
Confidence: High. Fabrication is architectural, not an instruction-following bug.
Why: Tolia's own anecdote shows a persistent system-prompt instruction failing, which matches every published result on hallucination: models produce fluent output whether or not grounding exists, and prompt-level "don't lie" text is a soft prior the sampler overrides when the answer is underdetermined. Frontier labs have shrunk hallucination rates with retrieval and verifier stacks, not with better instructions, because the problem lives in how the model generates tokens, not in whether it was told the rule. The opposite outcome, a model that obeys "never fabricate" from the prompt alone, would require solving calibrated abstention, and no lab is claiming that ships as a prompt-level fix in this cycle.
Revisit by 2027-06-01: We're right if the next frontier model still hallucinates on ground-truth-absent prompts despite an explicit no-fabrication system prompt, and production teams keep shipping RAG or verifier guardrails. We're wrong if a major lab ships a model whose system-prompt instruction alone drives fabrication to near-zero on adversarial ground-truth-absent queries in independent testing.
Comments