DeepSeek re-post-trained V4 Flash at the end of July. Same architecture, same 284B total parameters with 13B active per token, same weights budget. Only the post-training changed.
Here is what happened to the benchmark numbers, from DeepSeek's own published figures:
Benchmark Previous build 0731
SWE-bench Verified 79.0 79.0
DeepSWE 7.3 54.4
Terminal Bench 2.1 61.8 82.7
Cybergym 38.7 76.7
SWE-bench Verified did not move. Not by a rounding error, by zero. And DeepSWE went from single digits to the mid fifties.
If you pick models the way most of us do, by scanning a leaderboard for the highest coding score, this release looks like nothing happened. In an agent loop it feels like a different model.
What the two numbers actually measure
SWE-bench Verified hands the model a repo and an issue and asks for a patch. One shot. The model reads, reasons, writes a diff. It is a good measure of whether a model can write correct code.
DeepSWE and Terminal Bench measure something else. Can the model run a command, read what came back, notice the thing it expected did not happen, form a new hypothesis, and try again? Twenty times in a row without losing the plot?
Those are different skills. We have been treating one as a proxy for the other because the first one is easier to measure and has a longer history.
The retrain is clean evidence they come apart. The model's ability to write a function is unchanged. Its ability to operate a machine roughly quadrupled on one benchmark.
Why this matters more than it used to
Two years ago you pasted a function into a chat window and read the answer. Single-shot coding ability was the entire product surface. SWE-bench was a reasonable proxy.
Now most of us are running agents. Claude Code, Cursor's agent mode, Aider, Codex, whatever you have in your terminal. The model is not answering a question. It is running pytest, reading a stack trace, editing a file, running pytest again.
In that loop, a model that writes beautiful code but cannot recover from a failed command is worse than a model that writes mediocre code and reliably notices when it broke something. I would take the second one every day and so would you, after one afternoon with each.
The benchmarks most of us quote are still measuring the first thing.
The failure mode nobody benchmarks at all
While building our own terminal agent I went looking for how often agents just quit halfway. Not error out. Quit. Announce they are done when the task is visibly incomplete.
We had a completion check that fired when the model had produced a todo list. Reasonable design. Then I measured how often models actually produce a todo list on multi-step tasks.
13.6%.
So on the other 86%, nothing was checking. The agent said "done", the check had no list to compare against, and the loop ended. On cheaper models this happened constantly and it looks exactly like the model being dumb. It was not the model. It was a gate that almost never ran.
The fix was to make the gate unconditional and free, and to let a todo carry a verify command so that marking something done is a claim until the agent runs that command and sees it exit 0. Say it is done, prove it is done, then mark it done.
There is no leaderboard column for "quits halfway through". It is one of the largest practical quality differences between agent setups, and it is invisible to every benchmark you have read.
About that second table
The DeepSeek numbers above deserve a warning label, and I would be annoyed at myself for posting them without one.
Those are DeepSeek evaluating DeepSeek, on a harness that is not public, and two of the nine benchmarks they published are internal sets the company built. Take the direction seriously and the magnitude with suspicion.
The independent numbers, from Artificial Analysis who run the models themselves rather than republishing vendor claims, are more sobering:
GPQA Diamond: 90.8%
Humanity's Last Exam: 36.8%
AA-LCR, long context reasoning: 65.7%
CritPt, research level physics: 16.6%
AA-Omniscience non-hallucination rate: 15.6%
90.8% on graduate level science reasoning from 13B active parameters is genuinely impressive. That non-hallucination number in the same list tells you what you are dealing with. It is strong at structured reasoning and it will still assert wrong things with total confidence. Do not use it as an oracle. Use it as something you drive in a loop, with verification.
What to do instead of reading leaderboards
Nothing here is a case for ignoring benchmarks. It is a case for reading the one that matches what you do.
If you are pasting code into a chat window, SWE-bench Verified is still your number.
If you are running an agent, look for Terminal Bench, DeepSWE, and anything measuring multi-turn tool use. Those are the ones that predict whether the thing finishes your task.
And then test it yourself, on your repo, which is the part everyone skips. My rough test: give it a bug that requires running the test suite to find, in a codebase with a slightly unusual build step. Not "write me a function". Watch what it does on the third failed command. That is where models separate, and no benchmark I know of captures it.
The latency numbers matter here too, more than they do in chat. Measured through our gateway, median of three runs: 1.1s on a short prompt, 1.5s at roughly 6K tokens of context, 4.7s at 34K. Thirty-four thousand tokens of code answered in under five seconds is what makes something usable as an agent backend. A model that reasons brilliantly in 40 seconds per step is unusable in a 30 step loop, and that also does not show up on any leaderboard.
Disclosure: I build OpenAdapter, a gateway that serves DeepSeek V4 Flash among 40+ other open-source models on one API key, and Koda, the terminal agent where I found the 13.6% number. The measurements above come from running these models in production. Bring your own skepticism accordingly.
Top comments (0)