Lab-in-the-Loop at Compute Speed — Ian Quigley, Leash Bio

Full analysis

Leash Bio, a drug-discovery startup, ran a public test with roughly 4 billion binding measurements and 2,000 teams, and nobody beat a coin flip on genuinely new chemistry. Now they've open-sourced "Minos," a setup where language models learn chemistry by querying a database of pre-run experiments instead of a physical lab. The story that matters for anyone buying AI isn't the chemistry. It's the demonstration, in brutal detail, of how models fake competence by memorizing who made the data rather than learning the thing you paid them to learn.

How hard is this to undo? Nothing to undo. This is a lesson, not a purchase. The question is whether you change how you test the AI vendors and models you already rely on.

What's actually being decided: whether you trust a benchmark number from an AI-for-science vendor, or from any model trained on expert-generated data, without running your own out-of-distribution test.

What sets the deadline: nothing hard. The dataset is out now, which is the only live clock.

The Skeptic

The most useful thing Leash did was embarrass the entire field in public. 4 billion measurements, 2,000 teams, and on the hard split every single submission scored at chance. Three of the top five were the tutorial notebook. Read that twice. The winners weren't learning chemistry. They were memorizing shared building blocks. The chemist-identity result is worse: Ian Quigley's team predicted which of 1,815 chemists made a molecule from its structure alone, at 60% top-5 accuracy, and a representation built only from those chemist probabilities, with no chemical formula, predicted binding as well as the real molecule. The model was reading the author's signature. If you buy AI-for-science on a leaderboard number, you are buying a signature-reader.

The Researcher

The lesson generalizes, which is why operators outside biotech should care. Any model trained on data your experts generated learns your experts' habits. The underlying truth is a different thing entirely, and the score won't tell you which one the model found. Legal, medical, financial, underwriting. If three analysts each own a different class of deals and each has a style, the model can learn "which analyst touched this" and ride that to a flattering score. Leash calls it Clever Hans, after a horse that read its trainer's face instead of doing math. Their fix is instructive: they physically break the protein's binding pocket with a single mutation, re-screen, and train on the cases where a tiny change flips the answer. The business translation is to build a test set where surface cues are deliberately scrambled, so only real understanding survives.

The Open-Source Advocate

Minos runs on open-weight models, Qwen and GLM variants, fine-tuned to describe molecules in plain English. Not a frontier lab in sight. And the whole dataset, "Strelka," went out free on the day of the episode, following the BELKA data they already put on Polaris. This is the pattern that keeps repeating: the hard, useful work in applied AI is a domain-specific dataset plus a small open model, not a call to OpenAI. The 4 billion physical measurements one person generated in a basement are the moat. Anybody can download Qwen. Nobody else has the binding data. For buyers, that reframes what you pay for. You're paying for proprietary data and the test harness, and the model is the cheap, swappable part.

The Compute Pragmatist

The clever bit is cost, and it's worth copying. Leash turned a physical lab experiment into a database lookup. An agent gets 10 binders, 10 non-binders, and 100 queries against 6.5 million pre-screened molecules, and each query returns instantly because the experiment already ran. That makes reinforcement learning, where a model learns by trial and error against a reward, nearly free per attempt. They pulled over 10,000 structure-activity campaigns out of existing data in one weekend. A medic chemist sees maybe 50 in a career. The general pattern: if you can pre-compute the expensive feedback once and serve it from a database, you can run millions of training attempts for the price of a lookup. That's the reusable infrastructure idea, and it applies anywhere the "ground truth" is slow or costly to generate.

Where they disagree

The Skeptic says the field hasn't solved binding prediction at all, and most published scores are memorization. The Open-Source Advocate says that doesn't matter commercially, because the value was never the model, it's the data nobody else has. Both are right, and together they point somewhere uncomfortable for vendors: a high benchmark number tells you almost nothing, and a proprietary dataset tells you almost everything.

The second split is about how far this travels. Quigley pitches the plain-English reasoning as a platform play reaching into agriculture and materials science without new screening. The Compute Pragmatist's enthusiasm stops at the data boundary. The instant-lookup trick only works where you already screened the molecules. Step outside the 6.5 million, and you're back to slow, expensive physical experiments. The "generalizes everywhere" story is a hope. The "fast RL inside a known dataset" story is a demonstrated fact.

What it hinges on

One belief: does your model learn the task, or learn who generated the data? Leash gives you a cheap way to check it. Build a held-out set that shares no surface cues with your training data, the way they used a chemical library with no shared building blocks. If the score collapses on that set the way those 2,000 teams collapsed, your production number is fiction. Run that test before you renew any AI-for-science contract, and before you trust any model your own experts' data trained.

Prediction: When Leash Bio or an independent group reports results from the Minos / Strelka setup by the time of the next NeurIPS in December 2026, the plain-English language-model agents will still fail to beat chance on a genuinely novel protein-chemistry held-out split, the same failure the 2,000-team BELKA competition showed.

Confidence: Medium. The hard out-of-distribution problem is unsolved, and switching to language framing adds no new physical signal.

Why: Leash's own BELKA result showed that with 4 billion measurements and 2,000 teams, nobody beat random chance on a chemical library with no shared building blocks, because the models were memorizing surface features rather than learning binding physics. Minos changes how the model reasons, into plain English, but it draws on the same underlying data and adds no new physical measurements for proteins and chemistry it hasn't screened. Quigley himself argues language is a "regularizer" against memorization, a hope about reducing shortcuts, not a claim to have cracked generalization to new targets. The opposite outcome, a clean win on a truly novel split, would mean the core unsolved problem got solved by a change of output format, which is the least likely path given that every prior attempt on their own data failed.

Revisit by 2027-04-19: We're right if, by then, no published Leash or independent result shows the Minos-style agents beating chance on a held-out protein or chemical set with no shared building blocks or scaffolds. We're wrong if such a result is published and holds up on a genuinely out-of-distribution split.