NVIDIA research: agent harness beats model choice for long-horizon tasks

NVIDIA published research claiming the software harness around a model matters more than the model itself for long-horizon tasks. They wrapped Claude Opus 5 in their own supervisor-agent framework, called AVO, and pushed its score on the ARC-AGI-3 benchmark from 30% to 100%. NVIDIA VP of Product Adel El Hallack then used that result to argue open harnesses beat closed ones like Claude Code and Codex.

The core finding is real. A 30-to-100 jump on fixed weights is a genuine signal that capability on long-horizon tasks lives partly in the workflow scaffolding, not only in the model parameters. Microsoft's April research showing every frontier model corrupts documents on long edits, and Databricks CEO Ali Ghodsi's observation that the wrong harness can double your inference bill, both point the same direction.

The packaging is oversold, and the motive is obvious. NVIDIA builds and sells GPUs. A supervisor agent running alongside a primary agent means more round trips, more tokens, and more GPU-hours per completed task. Ghodsi's 2x cost warning is worth taking seriously precisely because the harness complexity NVIDIA is recommending runs on NVIDIA hardware. "Open harness" sounds like control until your security team asks who owns the memory layer and where tool-use gets logged.

The benchmark result itself is suspect. NVIDIA built the harness and ran the eval. ARC-AGI-3 is an interactive 2D game benchmark where a retry-and-backtrack supervisor loop can grind toward a perfect score in ways that don't transfer to your customer's messy PDF pipeline. A supervisor watching a primary agent also just moves the failure point up one level: now you need the supervisor to never drift, and there are two agents that can make a mistake instead of one.

The practical test, if you're building agents, is straightforward. Take one long-horizon workflow you already run, hold the model fixed, and run it three ways: bare, with a lightweight harness, and with a supervisor loop. Measure completed-task success rate and total tokens consumed. If the supervisor version doubles cost for a small accuracy gain, that's your answer.

The call: No independent team will replicate AVO pushing Claude Opus 5 to 100% on ARC-AGI-3 by the next ARC Prize cycle in December 2026. The outside score will land materially below 100%. Vendor-run, single-benchmark results almost never survive replication intact, and the track record on self-reported frontier milestones is consistent on this point. The harness-matters argument holds. The 100% won't travel.

NVIDIA published research claiming the software wrapper around a model matters more than the model itself for long tasks. They took Claude Opus 5, wrapped it in their own harness with a supervisor agent, and pushed a benchmark score from 30% to 100%. Then NVIDIA's VP of Product Adel El Hallack used it to argue open harnesses beat closed ones like Claude Code and Codex. For anyone building agents, the question is whether the harness is really the new lever, or whether NVIDIA just found a convenient way to sell more compute.

What's being decided: whether to treat harness design (memory, tools, feedback loops, supervisor agents) as a first-class engineering artifact with its own versioning and evals, or keep optimizing on model choice. Type 2, reversible. You can run a harness experiment on one workflow next sprint and roll it back. No deprecation forcing your hand. The only clock is your inference bill, and Ali Ghodsi's claim that the wrong harness can double it.

---

The Skeptic. NVIDIA built the harness and ran the eval. That is not replication, that is a vendor grading its own homework. ARC-AGI-3 is one benchmark built on 2D games with no instructions, which is exactly the setup where a supervisor loop that retries and backtracks looks magical. 100% on a puzzle-solving game does not tell me it holds on your customer's messy PDF pipeline. And a supervisor watching a primary agent just moves the failure up a level: now you're betting the supervisor never drifts. For a PM: NVIDIA showed a stunt double can nail one scene, not that the movie works.

The Compute Pragmatist. Ghodsi's 2x cost line is the real content here, and notice who's amplifying it. NVIDIA sells the picks and shovels. A harness that runs a supervisor agent alongside a primary agent means more round trips, more tokens, more GPU-hours per completed task. Their incentive is precisely to make you believe the answer to every problem is more inference. That said, the mechanism is right: supervisor round trips create idle GPU between subtasks, and your per-token pricing model stops predicting your bill once call topology dominates. For a PM: same engine, same trip, but a chattier co-pilot burns twice the gas.

The Researcher. 30% to 100% on the same weights is a real signal, not just a demo. It says capability on long-horizon tasks lives partly in the workflow, not only in the parameters. Microsoft's April finding that every frontier model corrupts documents on long edits, plus Ghodsi's cost observation, triangulate the same point from three directions. The open question is scaffold search: nobody knows the optimal harness topology, and a clean 100% on a game benchmark hides whether this generalizes. For a PM: the model is the engine, but how you chain its decisions may matter more than which engine you bought.

The Safety Lens. A supervisor agent does not solve alignment. It assumes a second agent is aligned and hopes the two never disagree in a way that breaks things. The AVO design hard-codes a chain of trust without saying how the supervisor's goals get verified. Open harnesses give you customization and control, El Hallack is right about that. But open surfaces also widen the attack area on tool-use and memory injection. Closed systems like Claude Code shrink surface area on purpose. For a PM: hiring a manager to watch your worker only helps if you trust the manager, and now there are two people who can make a mistake.

The Enterprise Buyer. No CTO signs a contract on one vendor-run benchmark. What I need is audit logs across both agents, a rollback story when the supervisor overrides the primary, and someone to indemnify me when the ensemble does something neither agent would do alone. "Open harness" sounds like control until my security team asks who owns the memory layer and where tool-use gets logged. The commercial subtext is that NVIDIA wants harness complexity to be my problem to configure and their compute to run. For a PM: the demo impresses procurement never.

---

Where they split. The Researcher and the Skeptic disagree on whether 30-to-100 is a finding or a stunt: same data, opposite reads, and the tiebreaker is independent replication that does not yet exist. The Compute Pragmatist and the Enterprise Buyer agree the mechanism is real but read NVIDIA's motive the same way: a company that sells GPUs telling you the answer is more GPU-hours. And the Safety Lens cuts against the whole "open beats closed" pitch: the thing that gives you control is the same thing that widens your attack surface.

What it hinges on. Two beliefs. One, does the harness advantage survive off NVIDIA's benchmark and onto real long-horizon work like code gen and document editing, run by someone who isn't NVIDIA. Two, does the supervisor-agent pattern actually reduce net cost, or does it trade a lower failure rate for a doubled bill, which is a wash for most workloads. The council leans toward "harness matters, but the specific NVIDIA framing is oversold." The core insight (scaffolding is a real capability lever) is sound. The packaging (open harness, NVIDIA-shaped) is a sales motion.

What to test before you retool. Take one long-horizon workflow you already run. Hold the model fixed. Run it three ways: bare, with a lightweight harness, with a supervisor loop. Measure completed-task success and total tokens, not per-token price. If the supervisor version doubles cost for less than a meaningful accuracy gain, you have your answer and it isn't NVIDIA's.

---

Prediction: No independent third party will replicate NVIDIA's AVO harness pushing Claude Opus 5 to 100% on ARC-AGI-3 by the next ARC Prize benchmark cycle in December 2026; the reproduced score under an outside team will land materially below 100%.

Confidence: Medium. Vendor-run, single-benchmark results almost never survive outside replication intact.

Why: NVIDIA built both the harness and ran the eval, and ARC-AGI-3 is an interactive game benchmark where a retry-and-backtrack supervisor loop can grind toward a perfect score in ways that don't transfer. The track record on self-reported frontier benchmark milestones is that outside teams reproduce a real but smaller effect once they control the harness and the prompt themselves, which is the whole reason the ARC Prize runs public verification. The opposite outcome, a clean independent 100%, would require the supervisor pattern to generalize perfectly across the exact conditions NVIDIA had a commercial incentive to tune for, which is the less likely case precisely because nobody outside NVIDIA has yet run it.

Revisit by 2026-12-31: We're right if an independent team (ARC Prize organizers or another lab) reports an AVO-style harness on Claude Opus 5 scoring below 100% on ARC-AGI-3. We're wrong if an outside replication confirms 100%, or if no independent replication is attempted at all.

The harness-matters point stands regardless. The 100% is the part that won't travel.