DeepSeek V5 is already getting plenty of attention, even though there still isn't much official information to build a serious technical evaluation around.
That's usually the point where model discussions start going in the wrong direction: people compare rumored parameter counts, expected context windows, leaked screenshots, and benchmark claims before there's even a stable API to test.
I'd rather use the time to prepare the thing that actually matters once the model ships: a repeatable eval harness.
Because when V5 does become available, the useful question won't be:
"Is DeepSeek V5 better than the previous generation?"
It'll be:
"Is it better for the work I actually run in production?"
Start with the model you already have
You don't need V5 to begin answering that question. Take the current DeepSeek generation you already use and turn it into your baseline.
For a coding workflow, that might mean collecting 30 or 50 tasks from real repositories rather than inventing synthetic prompts. A useful set could include things like:
Fixing a bug across multiple files
Adding a small feature
Understanding an unfamiliar module
Editing code without breaking existing tests
Working through a terminal-based task
Following tool output and recovering from errors
The important part is that these tasks stay unchanged. When a new model arrives, you run the same set again. That sounds obvious, but it's surprisingly easy to move the goalposts once you already know which model you want to win.
Don't stop at pass or fail
A model can complete the task and still be a bad production choice. I'd record more than just whether the final answer looks correct. For each run, track:
Task success
Latency
Input and output tokens
Number of retries
Number of tool calls
Failed tool calls
Human corrections
Total cost
For coding agents, keep close track of how often the model gets itself into unnecessary loops. A model that eventually solves the issue after 35 tool calls may look successful on a leaderboard, but it can still be worse to operate than one that finishes in 12.
The same goes for token usage. If V5 turns out to be significantly smarter but burns twice as much context on the same task, that changes the economics pretty quickly. This is why cost per accepted task matters far more than price per million tokens. A cheaper model isn't really cheaper if it needs three retries and a human cleanup pass.
Long context deserves its own test
This is another area where launch-day specifications can be misleading. A model might advertise a huge context window, but that doesn't tell you how well it uses information buried deep inside it.
If long context matters to your application, test it separately. Give the model:
A large repository
A long technical document set
Logs from multiple services
Several related files with conflicting information
Then see whether performance changes as the context grows. Test at a few different sizes rather than jumping straight to the maximum:
Plaintext
50K context
100K context
250K context
500K context
The point isn't to prove that the model can accept the tokens. The point is to see whether it can still find and use the right information.
What I'll care about when V5 actually lands
If DeepSeek V5 turns out to be a major upgrade, I expect the interesting improvements to show up in agent behavior more than in simple chat. I'll be watching coding, tool use, long-running tasks, and token efficiency much more closely than general question-answering benchmarks.
The workflow is straightforward:
Same tasks
Same prompts
Same tools
Same success criteria
Different model
Then compare the results. That also makes a unified API layer useful. If your application can keep the same request format while swapping models, you can test a new release without rewriting the surrounding infrastructure first.
CometAPI is one way to do that across different models and providers. The API layer stays mostly consistent, while the model becomes the variable you're actually trying to measure. That's especially useful during a launch, when the temptation is to change five things at once and then attribute the result to the new model.
Keep everything else boring. When V5 arrives, change one model ID, rerun the same workload, and see whether the numbers actually move. Until then, there's not much value in trying to benchmark a model that isn't available. There is plenty of value in making sure your benchmark is ready when it is.
Disclosure: This post is adapted from research originally published by the CometAPI team.
Top comments (0)