DeepSeek released V4.1-Flash-1017 on Oct 17, and the headline numbers moved:
Terminal-Bench 4.0 went from 31.2 to 42.3, and DeepSWE hit 77.0 — ahead of
Claude Opus 5's 74.0. The reaction split in two. Half the timeline celebrated.
The other half said what skeptics always say: show me the eval, not the vibe.
Both halves missed the interesting part. It was in DeepSeek's own release note:
the model is "tuned for DeepSeek Harness v0.2.1-rc.1 or later." Their
Terminal-Bench run was executed inside the Claude Code harness. DeepSWE ran
inside Mini-SWE-V2. The gains didn't come from a bigger model. They came from
the harness around the model.
The same model, five harnesses, five scores
DeepSeek published the comparison itself, on the official model card for
V4.1 Flash. Take one model and run it on Terminal-Bench 2.1 under five
different harnesses:
- DeepSeek Harness (Minimal): 90.6
- Claude Code: 88.0
- Pi: 86.1
- OpenCode: 85.0
- Codex: 84.1
Same weights. Same prompts, same tasks. A 6.5-point spread decided entirely by
the scaffolding the model runs inside. And when an independent tester, Vals AI,
ran the model on Terminal-Bench 2.1 under neutral conditions, they measured
74.53% — against the 90.6 vendor number. Another independent run put the model
through 10 real coding tasks in two harnesses and watched the token bill change
by ~75% depending on which harness drove it.
And it's not just the new model. The same pattern showed up on the previous
flagship: DeepSeek reported 87.9 on Terminal-Bench 2.1 for V4-Pro 0813; Vals
AI measured 54.68% on the reference harness — a 33-point gap. The harness
doesn't just nudge the score. It can move it by a third of the scale.
This is not a DeepSeek scandal. Every vendor reports the number from the
harness that flatters the model most. It's a measurement problem — and it has
a flip side nobody prices in: if the harness can cost a model 6 to 16 points,
the harness can also give those points back.
That's the bet I built on
Emma Intelligence started as exactly that bet: the bottleneck isn't the model,
it's the scaffold around it.
emma-scaffold-v3 is a scaffold and harness around an existing service — Gemini
Flash 4 — that allows it to perform better. On my benchmark suites it lifts
Gemini from 93% to 100%: 31/31 on reasoning, 31/31 on coding, 28/28 on math.
I don't expect anyone to take my word for it, and after this week you
shouldn't take anyone's. So I published the eval the way the skeptics are
asking for: every prompt and every verbatim model response, at
emma-intelligence.com/benchmarks. If you spot a bad grade in there, I'll own it.
The standard should be: show the eval
The lesson of the 1017 release isn't that vendor benchmarks are fake. It's
that a benchmark score is a property of a model plus a harness, reported by
whoever chose both. The fix is boring and total: publish the full table, name
the harness, and post the raw outputs. DeepSeek, to their credit, published
enough rope in their own scaffold table to make the harness effect visible.
Most vendors don't.
emma-scaffold-v3 wraps the base model with reasoning loops, verifiers, and
memory, so each answer is structured, checked, and grounded in context before
it reaches the user. The base model never changes — the work happens around
it.
What this doesn't claim
The scaffold doesn't make Gemini a different model. The 100% is on my
published suites — 90 hard questions, not the whole space of tasks. And the
harness effect cuts both ways: a bad scaffold can drag a good model down,
which is presumably why the neutral-harness numbers land where they do.
But the direction is settled. The model is becoming a commodity input. The
score lives in the scaffold.
Try it: emma-intelligence.com (free tier, 10 messages, no card).
The receipts: emma-intelligence.com/benchmarks
Top comments (0)