DEV Community: Haoxiang Li

Are You Benchmarking the Model—or the Harness?

Haoxiang Li — Fri, 14 Aug 2026 05:26:01 +0000

I nearly turned four software bugs into four model personalities

This article was edited with AI assistance. The cases and data come from real batch runs in the Kai! AI Arena. Every claim is limited to the game rules, model versions, and experimental setup used at the time. This is not a general model ranking.

If I had published one day earlier, DeepSeek V4-Pro might have acquired a personality trait it never had: bidding without looking at its dice.

First, a quick explanation of the table. In Liar's Dice, each player has a set of hidden dice. Players take turns claiming that the whole table contains at least N dice of a particular face. The next player must either raise the bid or challenge it. Everyone then reveals their dice: if the bid holds, the bidder wins; if it does not, the challenger wins. In Kai!, looking at your own dice is an explicit action, so a player may bid before looking. I call that a blind bid.

The evidence looked solid. In the first batch of AI matches, DeepSeek V4-Pro made nearly 40% of its bids before looking at its dice.

A story almost wrote itself. Perhaps it trusted intuition. Perhaps it was unusually willing to gamble or liked to seize the initiative. Another model usually looked first and calculated before bidding. Put the win rate, dialogue, and action logs side by side, and two distinct “personalities” seemed to emerge.

Then I inspected the context.

Even when the model had not used the probability tool, the system was still inserting a rough probability estimate into its prompt. The candidate actions were also sorted by probability. The model appeared to be bidding under the cup, but the harness was feeding it a strong hint from offstage.

After I fixed the leak, V4-Pro's blind-bid rate fell from roughly 40% to 6%.

What looked like a model insight had been an explanation of a software bug. A compelling conclusion vanished from the data.

Then I found three more bugs of the same kind. Across the first roughly 60 matches, more than half of the most visible differences between models shrank after the fixes.

That changed how I think about model evaluation:

An arena directly measures a system made of a model and a harness. Before attributing the result to the model, you must show that the measurement system did not quietly think for it, hide part of its input, or rewrite its failures.

By harness, I mean everything wrapped around the model: prompts, context assembly, tools, action spaces, token budgets, provider routing, output parsing, retries, and fallbacks. The model produces an answer. The harness decides what it sees, what it is allowed to do, and which part of that answer survives into the database.

The “personality” that fell from 40% to 6%

At first I thought I had found one implementation mistake. I kept looking and found a second, a third, and a fourth. The troubling part was that none of them stopped the matches. Every game still produced actions, dialogue, and a final score. The dataset looked complete.

Harness problem	The apparent model trait	What changed after the fix
`max_tokens=400` truncated long outputs	Poor formatting; often replaced by a fallback bot	Format failures and fallbacks dropped sharply
Rough probabilities were included without a tool call, and actions were sorted by probability	Liked to bid without looking	Blind-bid rate fell from about 40% to 6%
Opponent dialogue was not forwarded	Weak player with little social reasoning	Win rate returned to roughly 40–60%
Subjective judgments were stored only up to the first 100 characters	Rambling reasoning and incoherent records	The original chain of reasoning reappeared when full text was saved

The four bugs interfered at four different points.

The probability leak changed the input. Missing dialogue removed information the model should have had. The token cap truncated its output. The database then changed the evidence I used to interpret that output. The harness shaped both the move and my explanation of the move.

These distortions are especially dangerous because each one generates a plausible story. A truncated answer becomes weak instruction following. Missing dialogue becomes poor social reasoning. An action list that has already ranked the safe moves becomes decision-making ability. As long as the system still produces a score, it is easy to skip the measurement process and attach a label to the model.

The uncomfortable conclusion is that the evaluation framework is also playing.

So when I see a model leaderboard now, I do not begin with who won. I begin with a different question: What task did each model actually receive?

How can the same prompt become two different tasks?

Model arenas often point to a shared prompt as evidence of fairness. It is necessary, but it controls only one part of the experiment.

In the early version of Kai!, both seats received the exact same system prompt. Any of the following could still change the task:

the order of candidate actions;
whether the system supplied information the model had not requested;
whether the opponent's dialogue reached the context intact;
whether reasoning tokens and the final answer shared one budget;
whether a truncated answer counted as a failure, triggered a retry, or handed control to a bot;
which provider or quantized backend a model ID actually reached;
whether the parser saved the raw answer or only an excerpt.

A prompt is a string sent to a model. The task also includes its information boundary, tool permissions, compute budget, and failure policy.

This is why apparently uniform settings can create systematic bias. A concise model may escape truncation while a long-reasoning model spends its entire budget before producing a final answer. A model sensitive to option order will react to the placement of candidate actions. A model that relies heavily on conversational cues will lose more when dialogue is omitted.

Does a more uniform setup always make an experiment fairer? That depends on what you are trying to measure.

“Fair” is not a single configuration

Many arguments about benchmark fairness are really arguments about different questions.

If I already have a fixed product interface and want to know which model can replace another with the least work, I should hold the prompt, tools, budget, and parser constant. That measures compatibility with a shared product contract.

If I want to measure the ceiling of each model, I should optimize the prompt, tools, and reasoning settings separately. The result now includes adaptation work, so it no longer represents models running under identical conditions.

If I care about return on resources, I should fix cost, latency, or token use. That experiment measures output under the same constraint, not absolute capability.

All three protocols are valid:

Protocol	What is controlled	What it can answer
Interface fairness	Same prompt, tools, budget, and action space	Which model fits the same product contract best?
Capability ceiling	Model-specific prompt, tool, and reasoning optimization	What can each model do after adaptation?
Resource fairness	Same cost, latency, or token budget	Which model produces more under the same constraint?

Problems begin when the claim outruns the protocol. A shared-prompt benchmark can tell us which model works better with that prompt. It cannot directly establish each model's capability ceiling. Individually tuned results may show a ceiling, but the evaluator's tuning skill has entered the experiment.

Every benchmark report should state its fairness constraint first, then limit its claims accordingly.

The current “bare table” track in Kai! uses interface fairness. It answers a product question: if I swap the model behind the same game interface, what kind of opponent does the player get?

Once that protocol is chosen, the difficult work begins: keeping the harness from appearing in the score as model ability.

Keeping the harness out of the score

I turned that goal into a set of concrete constraints.

First, every model acts through the same player interface. It can see only its own dice and public events, and every action is checked by the same deterministic engine. The information boundary lives in the schema rather than in a promise written into the prompt.

Second, the same dice seed is played twice with the seats swapped. This reduces the effect of first move, seat position, and random rolls. A single win never becomes a model-level conclusion.

Third, I freeze more than the prompt. Candidate order, context serialization, tool responses, sampling parameters, parsers, and code versions are all experimental conditions. Each batch stores a prompt hash and Git commit. If one of them changes, I start a new batch.

Fourth, I record what the provider actually did. Sending max_tokens or a reasoning flag does not prove that the server honored it. Each call records completion tokens, reasoning tokens, finish reason, latency, cost, and actual route. Empty responses and timeouts remain in the dataset.

Finally, retries, repairs, and bot takeovers appear beside the score. A production system needs fallbacks, but an evaluation cannot let a fallback quietly impersonate model behavior. The leaderboard reports formatting failures, refusals, and fallback rates, with a separate view for zero-fallback samples.

These rules sound like engineering hygiene, but they decide whether the conclusion holds. A reliable product harness hides failures so the player can continue. A credible evaluation harness exposes them. The two can share code; they cannot share an unmarked data definition.

Even after all this, a leaderboard has boundaries. Controlling variables can make an answer more reliable. It cannot make the experiment answer a question it never asked.

How far can one leaderboard reach?

I divide the Kai! metrics into three layers:

Compliance: illegal actions, format failures, refusals, and fallback takeovers. Can the model fulfill the current interface contract?
Playing strength: win rate, successful challenges, and net chips. How good are its decisions under these rules and information conditions?
Behavioral texture: bluff rate, blind bids, raise depth, dialogue, and response rhythm. Does swapping the model create a perceptibly different opponent?

Each layer is useful, but its interpretation is limited. A low format-failure rate may come from better instruction following or a more forgiving parser. A high win rate shows strength at this game; it does not automatically become a general reasoning score. Behavioral differences matter to the product experience, but with thin samples they are observations, not permanent personalities.

The current clean set contains seven model IDs, 11 pairings, 22 seat arms, and 44 matches. That is enough to expose obvious harness failures. It is nowhere near enough for a general model ranking. The memory track also introduces cross-match path dependence, so it is stored separately from the memory-free bare-table track.

There is only one narrow claim I am willing to make from this dataset:

Under the recorded versions, routes, budgets, rules, and samples, different models behaved differently as opponents in Kai!.

One step beyond that requires another experiment.

Every model claim needs a counterfactual

The original story—“DeepSeek likes to bid without looking”—was easy to write. It had numbers, contrast, and the appeal of model personality. It was also false.

The rate moved from 40% to 6% without a model upgrade or prompt optimization. I merely removed information the harness had been leaking.

This is the part of model evaluation that leaderboards tend to hide. The model stands under the spotlight while the measurement tool disappears into the dark. Yet the tool still organizes the input, allocates the budget, handles errors, and decides which answer becomes evidence.

A perfectly neutral harness may be impossible. The design of the action space, the budget, and the metric set all involve choices. We can at least make those choices visible: pin versions, retain raw calls, publish failure rates, separate fallback samples, and state where the conclusion stops.

Now, whenever I am about to write “this model is bolder” or “that model cannot read people,” I ask one counterfactual question:

If I changed only the harness, would the difference survive?

Without that check, a model trait is only an observation waiting to be debugged.

So the next time a model looks brave, cautious, clever, or foolish, resist the personality analysis for a moment.

Check the table first.``

I Gave DeepSeek a Token Limit. It Ignored Me.

Haoxiang Li — Fri, 14 Aug 2026 04:03:20 +0000

A hands-on test of V4-Pro's default reasoning mode

⚠️ This article was generated with AI assistance (Claude). Every number comes from real calls the author made to DeepSeek's official API on the day of testing, and sample sizes are included with each conclusion. The author is an independent developer with no affiliation to DeepSeek or OpenRouter.

Tested on 2026-08-14 using deepseek-v4-pro on DeepSeek's official endpoint (the production V4-Pro-0813 release).

TL;DR

Reasoning is on by default. With no reasoning-related parameter, a single Liar's Dice decision consumed 3,072 output tokens—all reasoning, with no visible answer at all.
The OpenAI-style fix did nothing. I sent max_completion_tokens: 3072. The API returned HTTP 200, then generated 15,809 tokens over 222 seconds. It did not reject the parameter; it simply behaved as if it had never seen it.
Even a made-up parameter was accepted. totally_bogus_param: true also returned HTTP 200. Unknown parameters are silently swallowed, so you cannot tell whether a limit took effect until the bill arrives.
The same task took 2 seconds and 95 tokens with reasoning disabled. With reasoning enabled, it took 65 seconds and 4,500 tokens, while the visible answers were almost the same length.
Cost per usable answer: ¥0.0105 with reasoning off; ¥0.175—16.7× more—with reasoning on and an 8,192-token budget. With a 3,072-token budget, the effective cost was infinite: zero usable answers out of three, but every call was still billed.
Did quality improve? Against the older quantized build on OpenRouter, I found no detectable improvement on a hard metric with almost no room for interpretation (1/71 vs. 2/22; not statistically significant, with a small sample).
Reasoning can be disabled. reasoning_effort: "none" and thinking: {type: "disabled"} worked. enable_thinking: false did not—and the API never told me.

Why I Tested This

I am building a single-player Liar's Dice game called Kai! The opponent is not a scripted bot; it is powered by a large language model. It uses the same game engine as the player, sees the same information, reveals its thoughts before a challenge, and remembers how you played across rounds. Swap the model, and you effectively get a different opponent.

That means I need to know how much each decision costs and how long it takes. This was not benchmark curiosity. It was a practical problem forced on me by latency and billing.

Then I hit a wall.

The First Wall: I Paid for Three Empty Strings

The prompt for one Liar's Dice decision is about 3,200 Chinese characters. With max_tokens: 3072, three consecutive calls failed:

finish=length  completion=3071  reasoning=3071  visible=0 chars  50s
finish=length  completion=3072  reasoning=3072  visible=0 chars  47s
finish=length  completion=3072  reasoning=3072  visible=0 chars  42s
=> usable: 0/3

Every one of those 3,000-plus output tokens went into reasoning. Not a single character of the visible answer made it out. Billing is based on generated tokens, so I paid in full for all three calls and received three empty strings.

Worse, the application interpreted the result like this: no parseable action → fallback marks the move as noncompliant → the dashboard displays “this model disobeyed instructions in 94.4% of hands.” That figure came from my project's August 13 batch, not this isolated test. I nearly recorded a real token-budget failure as a model-behavior failure.

So How Much Budget Does It Need?

I ran the same task three times under four configurations:

Configuration	Usable	Completion tokens	Reasoning	Visible answer	Latency
Reasoning on · `max_tokens`=3072	0/3	3071–3072	100%	0 chars	42–50s
Reasoning on · 8192	1/3	7837–8192	~95%	0 / 118 chars	121–126s
Reasoning on · 32768	3/3	3220–5785	~97%	117–140 chars	49–82s
Reasoning off · 3072	3/3	84–107	—	123–156 chars	2–4s

Three things were true at the same time.

1. The tighter the budget, the more likely the model was to consume all of it. A 3,072-token budget ended at 3,072. An 8,192-token budget usually ended near 8,192. Only when I raised the limit to 32,768—far beyond the 3,200–5,800 tokens it normally needed—did it stop naturally and reliably.

2. Reasoning usage was hard to budget. On the same task, reasoning ranged from 3,137 to 5,691 tokens. With the intended limit ignored, it climbed to 15,774. This is not merely “thinking more.” It makes per-call budgeting unreliable: when usage can nearly double—or go much higher—which number are you supposed to provision for?

3. The longer reasoning did not produce a longer answer. With reasoning off, the visible answers were 123–156 Chinese characters. With reasoning on, they were 117–140. Forty-seven times the tokens and twenty-two times the latency did not produce a longer or more complete answer.

For my application, there was an even more fundamental problem: a round has a 30–60 second pacing budget, while one reasoning-enabled decision took 49–222 seconds. At that point, this was no longer just an issue of price. The product experience stopped working.

Is the Protocol at Fault, or Is DeepSeek?

Before publishing, I tested the parameter behavior separately. The answer is: both contributed, but not equally.

On the protocol side, the OpenAI-compatible max_tokens field places reasoning and visible output in the same budget. If reasoning consumes the allowance, the answer has nothing left. OpenAI's own reasoning models have had the same trap, which is why max_completion_tokens was introduced and why the documentation warns that a budget that is too small can produce an empty response. That part is a protocol-design problem.

Then I tested the relevant parameters against DeepSeek's endpoint:

Parameter sent	Intended behavior	Actual behavior
Nothing	—	Reasoning on by default; budget exhausted
`max_completion_tokens: 3072`	OpenAI-style completion limit	Silently ignored: 15,809 tokens, 222 seconds
`reasoning: {max_tokens: 1024}`	OpenRouter-style reasoning limit	Silently ignored
`reasoning_effort: "none"`	Disable reasoning	✅ Worked (18 tokens, 1 second)
`thinking: {type: "disabled"}`	Disable reasoning	✅ Worked (18 tokens, 1 second)
`enable_thinking: false`	Disable reasoning	Silently ignored
`chat_template_kwargs: {...}`	Disable reasoning	Silently ignored
`totally_bogus_param: true` (invented)	Reject the request	HTTP 200; silently swallowed

The final row is the root of the problem. The API silently accepts unknown parameters. Every ignored setting therefore looks exactly like a successful setting. You send a limit, receive a 200, and assume the call is capped—until the bill tells you otherwise.

max_completion_tokens is the clearest example. It exists specifically for reasoning models and for exactly the budgeting failure described here. DeepSeek accepted it, returned HTTP 200, and then let the model generate 15,809 tokens. Unsupported parameters can be rejected. Silently ignoring them is the worst possible behavior.

My conclusion: the protocol dug the hole; DeepSeek made it deeper and removed the ladder.

But Is the Production Release at Least Better?

This is where I most wanted the data to give DeepSeek some credit. It did not.

First, price needs careful wording. The older deepseek-v4-pro build on OpenRouter was listed at $1.17/$2.34 per million input/output tokens. The production -0813 release was listed at $0.43/$0.87, while DeepSeek's official endpoint charged ¥3/¥6—roughly $0.42/$0.85. On paper, the production release was about two-thirds cheaper. But OpenRouter had already priced the older deployment relatively high, and I cannot separate channel markup from an actual model price cut. So I cannot honestly claim that “the production release became more expensive.”

What did increase was the cost per usable answer. With reasoning off, one hand cost ¥0.0105. With reasoning on and an 8,192-token budget, only one of three calls produced a usable answer. Including the two wasted calls, the cost became ¥0.175 per usable answer—16.7 times higher. At 3,072 tokens, the effective cost was infinite: none of the calls could be used, but all were billed. Whatever happens to list price, this multiplier consumes the savings.

For quality, I chose a hard metric with almost no strategic ambiguity: when the current bid is already guaranteed to be true using only the model's own dice, the model still chooses to challenge. That challenge is guaranteed to lose.

Version	Relevant hands	Bad challenges	Rate
Older build (OpenRouter, quantized)	71	1	1%
Production 0813 (official endpoint)	22	2	9%

z=1.22; the difference was not statistically significant. The rigorous conclusion is not that the production release was worse. It is that I could not detect any improvement on this metric; if there was a directional signal, it pointed the other way.

This result needs an important caveat: the two versions were never tested head-to-head in the same batch. The comparison includes different opponents, random seeds, and prompt revisions. The production release also had only 22 relevant situations. The data can challenge a claim of obvious improvement, but it cannot prove regression.

Conclusion

I am not saying DeepSeek is bad. With reasoning disabled, V4-Pro returned a clean decision in two seconds for roughly one cent per hand, and it performed well at my table. My hosted seat still runs on DeepSeek.

The real problem is the combination of three product decisions:

Reasoning is enabled by default. For a per-call application, that can be a pure cost rather than a benefit.
There is no working reasoning-budget limit. Both the OpenAI-style and OpenRouter-style settings were ignored.
Unknown parameters are silently swallowed. This makes the first two problems hard to diagnose.

The third decision is the one that most needs to change. An API that returns an error can be debugged in ten minutes. An API that silently returns 200 forces users to work backward from a bill and a misleading “94.4% noncompliant” dashboard to discover what actually happened.

If you call this model programmatically, explicitly send reasoning_effort: "none" or thinking: {type: "disabled"} when you do not need reasoning. Do not rely on enable_thinking: false: in my test it had no effect, and the API did not say so.