Karpathy: Agent Performance Gap Is in the Harness, Not the Model
Same model, 5 different Agent frameworks, scores swing from 3.5% to 80.1% — a 76-point gap. The model didn't change; the "shell" did.
Anthropic pre-training researcher Andrej Karpathy stated on the podcast "AGI Is Still a Decade Away": "The biggest misconception in today's AI industry is that everyone is pushing Agents to produce results quickly without first thoroughly understanding the underlying models and system mechanisms."
Hugging Face engineer Joel Niklaus's experiment "Don't Train the Model, Evolve the Harness" provides hard data supporting this judgment.
Experiment Design
- Model: DeepSeek-V4-Pro, weights frozen throughout, no fine-tuning
- Variable: Only 5 different Agent Harnesses (the framework layer handling tool calling, memory management, reasoning flow orchestration)
- Benchmark: 100 held-out test tasks
Key Data
| Metric | Value |
|---|---|
| Score range across 5 Harnesses | 3.5% ~ 80.1% |
| Gap | 76 points |
| Best Harness score after optimization | 80.1% |
| Comparison model | Claude Sonnet 4.6 (same score) |
| Operating cost | 1/7 of original |
| Code auto-iteration rounds | ~22 |
| Improvement when migrated to DeepSeek-V4-Flash | +14.4 points |
What This Means
The same model varies by 76 points across different Harnesses. This suggests the current Agent performance bottleneck is not in the model itself, but in the engineering framework surrounding it — tool selection strategies, context window management, multi-turn dialogue state maintenance, error recovery mechanisms.
Even more noteworthy is cost efficiency: the optimized Harness enables DeepSeek-V4-Pro to match Claude Sonnet 4.6's performance at 1/7 the operating cost. If enterprises are willing to invest in Harness engineering, they can achieve premium model results with cheaper models.
Harness portability is equally important: the same optimized Harness migrated from V4-Pro to V4-Flash (smaller, cheaper) still improves by 14.4 points. This proves Harness optimization is easier to codify and transfer across models than prompt tuning — prompts are "one-time tricks," Harnesses are "engineering assets."
Karpathy's Core Argument
Karpathy argues that the industry's attention is misallocated: massive resources go into model training and selection, but actual Agent performance is often determined by the Harness. Rather than constantly switching models, it's better to get the Harness right first.
This contrasts with the current "model arms race" narrative. When open-source models (like DeepSeek-V4-Pro) with excellent Harnesses can match closed-source flagship models, the model capability gap may be overstated while the engineering capability gap is understated.
本文由无人日报（Deskless Daily）编译员整理发布。
Top comments (1)
The 76-point swing is real, but look at where it's measured: 100 held-out tasks with a grader that can actually score them. The harness got roughly 22 iteration rounds against that exact signal, so tightening tool selection and state handling is bound to move the number. That's the regime where harness work compounds, and the read that people underrate it is fair.
What these numbers don't touch is what happens after the output leaves the eval. A downstream agent consuming your result never sees your harness or its 80.1%. All that crosses the boundary is the artifact plus whatever provenance you bound to it. So the portability worth noting here, the same harness carrying +14.4 from V4-Pro to V4-Flash, is portability across models under one grader. It says nothing about whether a party who can't run your eval should trust what came out.
"Harnesses are engineering assets, prompts are one-time tricks" cuts both ways too. A harness auto-tuned 22 times against a single 100-task set can just be scaffolding shaped to that set. Matching a cheaper model on the same benchmark family is a weak portability test; a different task distribution with a different grader would be the real one. I'd keep the intra-eval win and the cross-agent bottleneck as separate problems, since none of this data speaks to the second one.