There is a new habit among people who use AI every day: comparing models the way we used to compare processors. GPT versus Claude. Claude versus Gemini. One benchmark table, another pricing table, another context-window table, and eventually a conclusion that sounds perfectly reasonable: this one is smarter than that one.
The problem is that we increasingly do not use bare models. We use systems built around them.
On August 21, Nvidia published a result that makes that distinction unusually visible. Its AVO architecture completed all 183 levels across the 25 public ARC-AGI-3 environments and scored 100 RHAE using Claude Opus 5 as the base model. A month earlier, ARC Prize had verified Claude Opus 5 at 30.2% on the same public benchmark.
Thirty versus one hundred looks like the kind of number designed to become a headline. It also needs a warning label. Nvidia itself says this is not a controlled comparison: reasoning settings, observation representation and parts of the evaluation setup differ. It would be misleading to say that “the harness added 70 points.”
But that is exactly why the result matters. The thing that changed was not just a better prompt. Almost everything around the model changed.
First: what exactly is a harness?
The word is not especially helpful. In AI, a harness is the software layer wrapped around a model that determines how it works: what context it receives, what tools it can use, what it remembers, how it recovers after an error, when it retries and what — or who — checks whether it is going off track.
If the model were a very capable analyst, the harness would be everything around the analyst’s desk. Access to the right documents. The history of what has already been tried. Excel. A browser. A terminal. A notebook that survives the end of a shift. A supervisor who notices that the analyst has spent two hours repeating an idea that is not working.
Remove all of that and you still have the same analyst. You do not necessarily get the same work.
Two products can advertise exactly the same model and still be selling very different systems.
What Nvidia changed around Claude
AVO — Agentic Variation Operators — was not originally built to solve games. Nvidia developed it for long-horizon engineering work, especially GPU-kernel optimization. In that setting, an agent cannot simply write a polished answer. It has to inspect code, propose a change, run it on real hardware, study the result, preserve what it learned and decide what to try next.
In one of those experiments, AVO worked for seven days, explored more than 500 optimization directions and ended with 40 kernel implementations. Nvidia reported improvements of up to 10.5% over FlashAttention-4 in the tested configurations.
The performance number matters less here than the duration. Seven days is an eternity for a model that normally receives a conversation, fills a context window and disappears. Keeping a process useful across hundreds of attempts requires solving a different problem: what should be remembered and what should be discarded.
AVO uses persistent memory to carry forward previous implementations, test results, compiler output and profiler data. It also uses a supervisor that watches the overall trajectory and can intervene when the agent falls into an unproductive loop. The main agent still makes decisions, but it is not working alone.
Nvidia then connected that same architecture to ARC-AGI-3.
A strange benchmark — which is why it is useful here
ARC-AGI-3 does not present a question and wait for an answer. It places the agent inside small interactive environments without instructions, explicit rules or a stated objective. The agent has to touch things, observe what changes, form a hypothesis about the world and revise it when it turns out to be wrong.
It is a bit like opening a strange video game with no tutorial. You press one key and something moves. You press another and discover that what looked like an obstacle is actually a door. The next level follows similar rules, but not identical ones. The challenge is not recalling an answer. It is maintaining an understanding that develops over time.
Claude Opus 5 was already strong when ARC Prize evaluated it in July. Its 30.2% was then the best verified ARC-AGI-3 result for a model. What became interesting next was how differently other architectures built around the same Claude behaved.
VISTA, a harness developed by MIT researchers, also completed all 183 public levels with Claude Opus 5 and scored 100. Its strategy was different: it gave the model PNG images of the environment and maintained lossless visual memory. Prime Intellect published three runs of Prime Agent with Opus 5; the median run reached 95.24% RHAE and the best reached 95.5%. Prime Agent uses a different approach again, including mechanisms that let parts of its working structure evolve while the task is underway.
Once several systems produce dramatically different results around a model we already knew, it becomes difficult to keep treating the model name as if it described the entire product.
This does not mean the model stopped mattering
It is worth avoiding the pendulum swing. A few months ago the easy claim was “the model is all that matters.” The equally easy reaction now would be “the model no longer matters.” Both are too convenient.
A harness cannot manufacture reasoning ability from nothing. If the model cannot interpret an observation, use a tool or maintain a coherent hypothesis, adding memory and a supervisor may simply produce a better-organized system that still cannot solve the task.
Nvidia is also explicit that its 100 score and ARC Prize’s 30.2% are not a scientific ablation where every variable is held constant except one. The AVO run uses a different reasoning configuration and a text-based 64×64 grid representation, while other systems use images. Memory, context management and execution loops also differ.
That caveat does not weaken the larger point. It strengthens it. If all those surrounding pieces can move the outcome, then measuring only the model tells us less and less about the system a user will actually experience.
Real-world comparisons are already contaminated by the system around the model
Imagine you are choosing an AI tool to review contracts. You test two products. Both say they use Claude Opus 5.
In the first, you upload twenty PDFs. The system extracts the text, builds an index, retrieves relevant passages, keeps a persistent memory of the case and forces the model to cite the page behind each claim. If it detects a contradiction, it runs a second pass before showing you the answer.
In the second, you upload the same files. The application pushes as much text as it can into context, summarizes the rest and calls the same model once.
Both can put “Powered by Claude Opus 5” on the pricing page. Comparing them as “Claude versus Claude” explains almost nothing.
Something similar already happens in office workflows. In our earlier analysis of AI and decision-making, we described supervision debt: the work that appears after automation, when someone has to verify sources, correct interpretations and understand why two tools reached different conclusions. Some of that debt can come from the model. Some comes from the workflow built around it.
That also explains a familiar frustration: you switch to a supposedly better model and your favorite product does not improve by the same amount. The benchmark moved. Your experience did not necessarily move with it.
The better question is no longer “which model does it use?” but “what does it do with that model?”
For most users, “harness” can remain an unnecessary word. What matters is translating it into practical questions.
- Does the tool have persistent memory, or does every task begin almost from scratch?
- Can it search, run code, open files or query databases?
- How does it decide what enters the context window and what gets left out?
- Does it check its own work before returning it?
- Can it recover when a strategy fails, or does it simply retry the same idea with different wording?
- How much of the cost comes from the model itself, and how much comes from all the additional calls around it?
That last question deserves more attention. A good harness can improve results, but it can also multiply model calls, context and execution steps. Prime Intellect publishes cost and token-use curves for exactly this reason. The problem is no longer just “how smart is it?” It is also how much it costs to turn that intelligence into a completed task.
In other words, a more capable system can also be slower or more expensive. The best architecture depends on the job, not on the excitement around a benchmark.
It also changes how we should think about lock-in
There is a quieter business consequence. Until now, many companies have treated dependency risk as a provider question: “What happens if we want to leave OpenAI, Anthropic or Google?”
But if real performance increasingly depends on memory, tools, evaluators, retrieval, permissions, internal formats and supervision logic, the hardest part to move may not be the model. It may be everything wrapped around it.
That cuts both ways. A company that owns its orchestration layer may be able to swap models with less disruption. A company dependent on a closed product may discover that “it uses Claude” describes only a fraction of what it actually bought.
Strangely, this can reduce and increase lock-in at the same time. It reduces dependence on the base model if the harness is yours. It increases dependence on the vendor if the vendor’s harness is the reason the product works so well.
We may need benchmarks for products, not just models
Model rankings remain useful. They tell us something about base capability and allow comparisons under relatively known conditions. The problem begins when we take that number and project it directly onto a real application.
A real application includes search, memory, permissions, files, tools, system instructions, retries and cost limits. Sometimes a second model checks the first. Sometimes deterministic software does the important part while the LLM merely decides when to run it.
That is why the unit we eventually compare may not be “GPT versus Claude versus Gemini.” It may be complete systems doing complete jobs: how long they took, what they cost, how often they needed human intervention and what they did when something failed.
That is less convenient than a leaderboard with one big number beside each model. It is also much closer to the way people actually use AI.
The model is still the engine. But we are now driving the whole car
Nvidia’s result does not prove that Claude Opus 5 becomes three times better when someone gives it memory. Nor does it prove that model benchmarks are obsolete.
It shows something more uncomfortable: a growing share of the capability we attribute to “the AI” emerges from the interaction between the model and the system around it.
That should change a habit. When a new product says it uses the most advanced model on the market, the evaluation should not end there.
What does it remember? What can it do? How does it check its work? What happens when it is wrong? What does that whole process cost? And how much of the result would disappear if you changed the model tomorrow but kept everything else?
We may eventually discover that we spent months arguing about the best engine without looking at the vehicle it was mounted in.
References
Nvidia describes AVO, its persistent memory, supervisor, kernel-optimization results and ARC-AGI-3 evaluation. Read Nvidia’s technical post.
ARC Prize published the verified Claude Opus 5 result of 30.2% on ARC-AGI-3 and documents the benchmark methodology. See the ARC Prize results.
The MIT VISTA team documents how its visual harness with Claude Opus 5 completed all 183 public ARC-AGI-3 levels. See VISTA.
Prime Intellect describes Prime Agent and publishes three Claude Opus 5 runs, including a 95.24% median RHAE run and a 95.5% best run, alongside cost and token-use data. See Prime Agent.
