Ex-OpenAI Researcher Claims Neural Networks Are Already Interpretable

Full analysis

A former OpenAI researcher just claimed the inside of a neural network, the tangle of math that makes a model work, is far easier to read than the field thought. If true, it means the circuits that do a model's thinking can be inspected directly, so a model that reasons one way inside and talks another way outside has nowhere to hide. That's the whole game for AI safety. It's also an unpublished essay from one author who puts his own confidence at 85%.

What's actually being decided here isn't whether Nick Cammarata is right today. It's whether teams that build with and buy AI should change anything now, before the peer-reviewed version lands. This is easy to undo. Nobody has to rip out a stack. Nothing forces a deadline except the paper itself, which hasn't dropped.

The Skeptic. The headline should be the 85% confidence on an unpublished, single-author essay claiming a field-wide paradigm shift. Superposition, the idea that a network crams several unrelated concepts into one detector, didn't come from careless people. It came from Anthropic and DeepMind researchers staring at real evidence that detectors fire for more than one thing. Cammarata re-reads one famous detector as "pointed shapes" instead of "cats and cars." Cute. It doesn't erase the pile of other evidence. And "9 out of 10 detectors have one primary job" is not the same as "the circuit is clean." A primary job can sit on top of a mess that just moved one layer down. The leap from legible detectors to legible reasoning is the part he's hand-waving.

The Safety Lens. If Cammarata is right, the prize is specific: deceptive reasoning becomes catchable. The scenario that keeps alignment people up at night is a model that schemes internally while producing polite, aligned-looking answers. Readable circuits would let an inspector see the gap. But notice the phrase in his own quote: "solved by the time we need it most." That's a timeline claim with the stakes of the century attached, made before review, and "when we need it most" is a threshold nobody has defined. Neel Nanda, who runs interpretability at Google DeepMind, called it interesting but "not a significant update to the big picture." That's the right posture. This is a reason to invest more in reading model internals, not a reason to relax.

The Researcher. The methodological claim is worth taking seriously, and that's different from accepting it. If superposition was partly an artifact of blurry visualization tools run on a badly-chosen 2014 model, Google's Inception v1, that's a real scientific correction. The gap he reports is stark: about 2% of circuits were readable in the 2023 work where GPT-4 tried to explain GPT-2, versus 9 out of 10 in a modern model. A delta that big demands replication, not a shrug. But the test is whether legibility survives where it's hard: deep in the network, inside attention heads mid-circuit, and under adversarial inputs designed to break it. Clean detectors on the outer layer are the easy case. Nobody has shown the hard case yet.

The Compute Pragmatist. One claim nobody's pricing: Cammarata says bigger models are easier to read, because they can afford to spend neurons on one idea each. That flips the standard safety story on its head. The going assumption has been that scaling up makes alignment harder because the internals get murkier. If legibility actually improves with scale, the "stay small to stay safe" argument weakens, and the case for doing interpretability work at full frontier scale gets stronger. It also means years of interpretability research done on tiny proxy models may have been studying the wrong thing. That 2%-on-GPT-2 figure isn't just embarrassing. It suggests the small-model work was systematically misleading about what big circuits look like. If that holds, research compute gets reallocated toward reading large models directly.

The Builder. Forget the philosophy. What changes on Tuesday? Today, auditing a model is behavioral. You poke the outputs and squint at the results. If circuits are genuinely readable, safety teams could write real tests against what's happening inside the model, not vibes-based checks on what comes out. But the bottleneck moves. Reading a circuit once in an essay is one thing. Instrumenting that reading into the training pipeline, so you catch a bad circuit as it forms, is plumbing nobody has built. The teams that start wiring interpretability hooks into their own training now get a head start when the paper lands and everyone piles in. The ones waiting for a turnkey tool will be waiting a while.

Where they split

Three real disagreements. The Compute Pragmatist sees a clean inversion, bigger is more legible, that would reshape where safety money goes. The Skeptic thinks "legible detectors" and "legible reasoning" are being quietly welded together when they're not the same claim at all. Both can't be fully right.

The Safety Lens and the Researcher agree on posture, Nanda's caution, but for different reasons. Safety wants it true and is guarding against wishful thinking. The Researcher just wants the hard cases tested before the priors move.

And the Builder is the odd one out: he doesn't need the grand claim to resolve. Even a partial win gives teams something to build. That's the most grounded read in the room.

What it hinges on

One thing. Does legibility hold where it's hard, deep layers, mid-circuit attention heads, adversarial inputs, or only on the outer-layer detectors that were always the easy case? Cammarata's essay shows the easy case beautifully. The field's existing evidence lives in the hard case. Until an independent team reproduces the 9-out-of-10 figure on the hard case, this is a promising essay, not a correction to the record. The council leans skeptical-but-interested: real enough to watch the replication, nowhere near solid enough to change a safety budget on.

What to do before committing anything: don't. If you build with AI, this changes nothing about your stack this quarter. If you run a safety or audit team, the cheap move is to start building the pipeline plumbing the Builder named, because that's useful even if Cammarata is only half right.

The call

The gap between Cammarata's framing and what independent review will support is the thing to bet on. He says "solved soon." Nanda says "not a significant update." Those can't both survive replication intact, and the structure of these claims tells you which way it breaks. Single-author, pre-publication, paradigm-shift essays in a field with contested ground truth almost never land fully intact. They land as "real effect, narrower than claimed." The superposition framework came from people looking at genuine multi-concept detectors, and one reinterpreted example doesn't vacate that evidence.

Prediction: When Nick Cammarata's peer-reviewed interpretability paper is published, the independent follow-up work it triggers will confirm a real legibility effect but will NOT support his "ambitious interpretability will be solved soon" framing, with at least one interpretability group (Anthropic, Google DeepMind, or an academic lab) publicly reporting that superposition remains a live phenomenon in modern models.

Confidence: Medium. The effect is probably real. The "solved" framing almost never survives adversarial replication.

Why: Cammarata reports 9-out-of-10 legible detectors but rates his own confidence at 85% on an unpublished single-author essay. His strongest evidence, the "pointed shapes" reinterpretation, addresses outer-layer detectors. The deep circuits and attention heads where the field's superposition evidence actually lives remain untested by his work. Neel Nanda, who leads interpretability at Google DeepMind and has no incentive to undersell good news, already called it "not a significant update to the big picture," which means the field's center of gravity hasn't moved. The pattern for grand pre-publication claims in contested fields is consistent: they resolve as a genuine but narrower effect once other groups try to break them. For the opposite to happen, the entire superposition evidence base would have to be an artifact, and one reinterpreted detector doesn't get you there.

Revisit by 2027-04-23: We're right if, after Cammarata's paper publishes, an independent interpretability group publishes work confirming a legibility improvement while reporting superposition or polysemantic detectors still present in modern models. We're wrong if independent replication broadly endorses the "interpretability is largely solved" claim with no major group reporting surviving superposition.