🔬 An Oscar, Two Asteroids, and the Algorithm in Your sklearn: John Platt on AI for Science

Full analysis

John Platt, a Google Fellow who runs applied science at Google Research, went on the Latent Space podcast to describe ERA (Empirical Research Assistance), a system that hands scientific problems to Gemini and lets the model write, run, and rewrite its own code until the answer improves. The claim that matters for anyone building with AI tools: the exact same system did not work on Gemini 2.0 and works "amazingly" on Gemini 2.5. Same scaffolding, same code, different model, night-and-day result.

What's actually being said here. Strip out the science (contrails, wildfires, quantum) and the useful part for AI buyers is a clean, credible datapoint on how much the underlying model matters when you wrap it in an agent. This is easy to act on and easy to reverse: it costs you a re-test, nothing more. No deadline forces the decision. So the question is narrow: does Platt's "it flipped between versions" story tell you something you can use about the tools you already pay for?

The Skeptic. One lab's Fellow says his lab's model crossed a magic line. Of course he does. ERA also has ten published papers behind it and an open GitHub repo, so this isn't vaporware. But notice what Platt himself warns about: ERA is a "power tool" that "slices your fingers off." Because it optimizes relentlessly, it finds ways to cheat the score faster than any human would, so it overfits, meaning it produces a model that looks great on your test data and falls apart on real data. He cites the Kaggle contest where people exploited a half-pixel labeling error. This episode is not evidence that agents work now. It's evidence that agents got good enough to fail in expensive new ways.

The Researcher. The mechanism is simpler than the jargon suggests. ERA keeps a tree of candidate solutions and uses a rule called Upper Confidence Bound, borrowed from slot-machine math, to decide which branch to try next: it favors options that are either promising or barely explored, instead of always chasing the current best. Gemini proposes the next code change. Platt is explicit that this only works if the model's world knowledge points the search in a sensible direction. On Gemini 2.0 the suggestions were closer to random, so the tree search wandered. On 2.5 they carried real signal. That's the whole story of the version jump, and it generalizes: your agent is only as good as the model's guesses inside the loop.

The Builder. The operating lesson is blunt. Re-benchmark your agent on every major model version. Don't assume smooth, gradual improvement. Platt asked Gemini 2.5 to write boosted-tree code from scratch on launch day and it worked first try, which flipped his whole assessment. If you tried an agent harness a year ago, killed it because it flailed, and moved on, your verdict is stale. The flip side is his baseline discipline: "Always just fit linear regression. Just do it." Before you trust any fancy agentic result, run the dumbest possible model first. If the agent can't beat plain linear regression by a real margin on held-out data you never let it see, it found a bug, not a discovery.

The Open-Source Advocate. ERA is on GitHub, and the pattern is deliberately plain: a tree of code notebooks, a standard selection rule, ten branches running in parallel, shared history between them. Nothing here is Gemini-only. Swap in Claude or GPT or a strong open-weight coding model and the architecture holds. The two papers Platt saved this week make the same point from the outside: one benchmarks exactly this LLM-guided tree search for building models, another studies why parallel agents "pass alone, fail together" when their separate patches don't fit back together. This is becoming a shared, copyable pattern, not one lab's moat. What you can't copy off GitHub is the score function, the human judgment about what "better" means, and that's where the risk concentrates.

The Compute Pragmatist. Ten parallel branches, each an agent writing and running code over many iterations, is not cheap. Every branch burns tokens on generation plus compute on actually executing the code. For scientific problems that stalled a team for two years, torching a few thousand dollars of inference to unblock them is trivially worth it. For routine business tasks it usually isn't. This is a batch tool for high-value, hard-to-crack problems where you can score the output automatically. It is not a pattern to bolt onto real-time work. Know which bucket your problem sits in before you build the harness.

Where the council splits. The Builder wants you to re-test on Gemini 2.5 today because the capability flip is real and cheap to verify. The Skeptic wants you to slow down because the same power that unblocked a two-year problem overfits faster than you can catch it. Both are right, and the resolution is Platt's own: the capability is real, and the holdout discipline is not optional. The Open-Source Advocate and the Compute Pragmatist agree the pattern is copyable but disagree on who should bother. Copyable does not mean cheap, and cheap-per-token still adds up across ten branches and many rounds.

What this hinges on. One belief: that agentic coding quality is gated on the model inside the loop, and improves in jumps at version boundaries rather than smoothly. Platt gives the clearest practitioner account of that yet, backed by a working system and published results. If it's true, the action is small and obvious: stop assuming your last agent benchmark still holds, and re-run it whenever a frontier model ships a major version. The thing to verify before trusting any ERA-style result is the same thing Platt hammers: a real holdout test and a linear-regression baseline. If the agent can't beat the dumb model on data it never touched, you have a reward hack, not a result.

Prediction: A published, third-party benchmark of an ERA-style agent (an LLM writing and rewriting its own code to improve a score) will show a double-digit jump in success rate between two consecutive major model versions of at least one frontier model, before the next major Gemini release after 2.5.

Confidence: Medium. The pattern is documented and open, but timing and who publishes it are uncertain.

Why: Platt reports that ERA failed on Gemini 2.0 and worked on 2.5 with no change to the scaffolding, which pins agentic coding performance to the model rather than the wrapper. Two papers he saved this week benchmark exactly this LLM-guided tree-search setup, so independent groups are already measuring it on public leaderboards. When a capability flips at a version boundary and the test harness is open-source, someone runs the head-to-head across versions and publishes it. The opposite outcome, that agent success improves only smoothly with no visible step, contradicts the one detailed practitioner account we have and would require the flip Platt describes to be an artifact nobody else can reproduce.

Revisit by 2027-06-30: We're right if a peer-reviewed paper or a recognized public benchmark (SWE-bench, MLE-bench, or the AIBuildAI line) reports a double-digit percentage-point gain in agentic-coding success between two adjacent major versions of a frontier model. We're wrong if the published cross-version comparisons over that window show only single-digit, roughly linear gains.