DEV Community: Haoxiang Li

Codex vs. Claude Code at Liar's Dice: the Winning Bluff Was the Truth

Haoxiang Li — Mon, 17 Aug 2026 15:14:49 +0000

One authoritative engine, two seat-locked MCP servers, three best-of-threes, and a 3-millisecond whodunit

The matches are real: Codex CLI (gpt-5.6-sol) against Claude Code (Claude Opus 5), both playing through the same rules engine. Every number below was recomputed from the raw run.json and both session logs, and every game replays deterministically from its seed. Quotes from the agents are verbatim from decision-time records. None of this is a general model ranking.

I wired Codex CLI and Claude Code into the same Liar's Dice engine over MCP and had them play three best-of-3 series. Claude won all three, 2–0 each time. Its challenge calls hit 8 out of 11; Codex's hit 4 out of 26.

The score takes two sentences. The parts worth writing down took longer: how to build a table that two closed-source agents can't cheat at, two numbers that surprised me, and an incident where I almost blamed a model for something its CLI did.

The table

Liar's Dice in sixty seconds: five dice each, and you only see your own. Players alternate bids of the form "there are at least N dice showing X across the whole table." On your turn you either raise the bid or challenge it. On a challenge everyone reveals; if the bid stands, the challenger loses a die, otherwise the bidder does. Run out of dice and you lose the match. Ones are wild by default.

The rules are the easy part. The hard part is making the result trustworthy. Codex and Claude Code ship with their own system prompts and tool loops, so the referee has to guarantee three things by construction: neither side can see the other's dice, the referee has no side channel that favors anyone, and the "what it was thinking" quotes you read afterward were actually written at decision time.

The setup is one in-process rules engine behind a localhost-only HTTP coordinator, with two stdio MCP servers doing nothing but forwarding:

 Codex CLI (gpt-5.6-sol)       Claude Code (Opus 5)
       |  stdio MCP                 |  stdio MCP
       v                            v
  [seat-mcp A] --token A--+  +--token B-- [seat-mcp B]
                          |  |
                          v  v
              +------------------------+
              | coordinator @127.0.0.1 |
              |  - createMatch engine  |
              |  - stateId concurrency |
              |  - run.json audit log  |
              +-----------+------------+
                          | SSE
                          v
                    [ /spectate ]

The load-bearing decisions:

Tokens bind to seats at process start. The tool schema has no seat parameter. A client that wanted to impersonate its opponent would have nowhere to type that.
Information hiding lives in the schema. A seat's view reuses the game's observe() projection, and that JSON has no field for opponent dice. Removing the field beats writing "please don't peek" in a prompt.
Actions use optimistic concurrency. Every observation carries a stateId; submit against a stale one and you get a 409. Concurrent peeks produce a handful of these per run (0–7), all preserved in the rejection log.
Thinking is recorded at act time. Every action must include a private belief and may include one public say. The belief commits atomically with the action; the opponent never sees it and nobody can rewrite it afterward. Every quote below comes from there.
Latency is coarsened. Raw milliseconds go only to the audit file; seats and spectators see "fast" or "slow." Without this, response time is a usable side channel.
Dice go through commit-reveal. Each round opens by publishing hashes of both hands, and the reveal has to match them. The referee couldn't quietly reroll dice if it wanted to.
A challenge carries its meaning. Challenges must include assert: "current_bid_is_false" or they're rejected. This came out of an earlier replay study: for one model, turning "challenge" from a bare verb into an assertion it has to type out cut guaranteed-loss challenges from 23% to 2%. Tool schemas change behavior, and that one is measured.

Both seats receive word-for-word identical instructions and the identical task prompt, each running in an isolated temp directory with user config ignored. One session plays the entire series, so cross-game memory is part of what's being tested. What I can't control is each CLI's internal prompting and scheduling, which is why the contestants are, and stay, two systems: Codex+Sol and Claude Code+Opus. Every claim here is scoped to that.

Why trust the numbers

After the runs I did four checks:

Deterministic replay. The engine is a pure function of seed and action sequence. A verifier takes each run's seed plus its accepted actions, replays all events, and diffs them item by item against the archive. Five runs, 8 games, 432 events: all identical, timing fields included.
Every rejection is logged, with the raw action and a timestamp. Illegal actions never touch the game state, but they never vanish from the record either. Section five leans on this.
Model identity comes from receipts. Every Claude gameplay response reports claude-opus-5; tokens, cost, and cache reads for both sides are archived.
Directories are frozen with SHA-256 manifests.

The workflow:

# one continuous-session BO3; the mirror run just flips --codex-seat
node scripts/mcp/run-showdown.mjs --best-of 3 --seed 73019426 \
  --codex-seat A --codex-model gpt-5.6-sol --claude-model opus

# verify later: replay all events from seed + actions, diff against archive
node scripts/mcp/replay-showdown.mjs docs/showdown/<run-dir>

Results

Run	Setup	Series	Sol challenges	Opus challenges
E0	Sol in seat A, continuous session	Opus 2–0	0/8	2/4
E1	same setup, fresh seeds	Opus 2–0	2/10	2/2
E2	E0's seeds, agents swap seats	Opus 2–0	2/8	4/5

E2 is the control that matters. It reuses E0's seeds exactly and only swaps which agent sits where. All four paired games followed Opus rather than the seat, which crosses off "seat A is better," "going second is better," and "that seat got luckier dice." To be precise about what the mirror controls: the random stream is anchored per seat, so round-one hands match the original run die for die, and later rounds drift as dice counts diverge.

Then there's the stat that made me stop. Across the three series, when Codex challenged one of Claude's bids, the bid was true 22 times out of 26. When Claude challenged Codex, the bid was false 8 times out of 11. In a bluffing game, the side doing most of the truth-telling was the one running the traps.

Where Sol lost

Sol's strategy reads straight out of its own logs: peek; open on your longest suit at "own count + 1"; when a bid comes in, compute a binomial probability that it stands; challenge below a threshold. Clean, stable, locally correct at every step.

The flaw is in what the computation assumes about the opponent. It answers: if the opponent's dice were uniformly random, how likely is this bid to stand? But the opponent's bids aren't random draws. Opus only pushed a count high when its own hand already covered most of it, so the fact that Opus chose that bid carries information. Sol's formula has no term for that, and it kept converting correct arithmetic into wrong decisions.

Opus worked the gap methodically. During game one it decoded Sol's openings ("longest suit + 1" amounts to announcing your hand). Then it learned Sol's challenge threshold. Then it started manufacturing bids that look suspicious under the random assumption and happen to be true. The worst stretch ran three consecutive rounds: a true bid placed right at the edge of standing, a challenge, a lost die, three times over. In the middle of it Opus said, publicly: "Three fives. You skipped past my fives twice now instead of testing them — I don't think that's an accident." A true statement about a true bid. Sol challenged anyway.

Opus wasn't reading hidden dice, and it wasn't infallible. It challenged one of Sol's borderline-true bids and paid a die for it. Its opponent model was wrong now and then; it kept updating regardless.

The cross-game gap is visible in the logs too. Counting explicit references to game one inside game-two decision records: Opus made 5, 2, and 3 across the three series. Sol made zero in all three. Both sides had the same continuous session. One of them used it.

The 1,093-call whodunit

The mirror run had an incident. In two opening states Sol submitted a bid with count 1 — the floor is 2, in both the rules and the tool schema — and after the rejection it submitted the same action again. And again: 1,093 times across the two states.

My first reading was "the model is melting down." The rejection log says otherwise. Pull the timestamps and 601 of those rejections land inside 22 seconds, median gap 3ms; the other 492 land inside 12 seconds, also 3ms. A model forward pass takes hundreds of milliseconds at best, so nothing was deciding forty times a second. That cadence is a retry loop. The independent-seed series provides the cross-check: the same count-1 mistake appeared there exactly once, got rejected once, and was corrected on the next call. So the ledger splits: writing an illegal bid was the model's error; repeating it six hundred times was the CLI's retry machinery.

Without per-rejection logging, the aggregate line "Sol submitted 1,093 illegal actions" would have read as model behavior, and it would have made great copy about a panicking AI. I've been burned by this category before — my earlier piece on harness-vs-model attribution came out of four bugs that had all been masquerading as model personality. The question worth running before any "the model is X" claim: swap only the harness — does the behavior survive?

The incident also exposed a debt in my coordinator. It needs a circuit breaker: collapse repeated identical rejections, back off, flag the sample as contaminated. It held up under six hundred hits this time. It shouldn't have to next time.

The ledger

Two systems played, and the biggest confound is compute. Opus generated about 300k output tokens across the three series; Sol generated 19k. That is a 16x gap, $17.63 in receipts on the Claude side. How much of the win is better conditioning and how much is simply more thinking, I can't separate yet.

The sample is small. Six games across three series, and the two games inside a series aren't independent, because cross-game memory is the mechanism under test.

One line points the other way. The very first match I ran used Codex's default model — not Sol, no best-of-3 — and it was close: 5–4 in rounds. Every blowout happened under the combination "fixed strategy × continuous session," which reads like the gap was learned during play rather than innate. That's a testable claim, not a conclusion.

The experiment queue, in order: restart sessions every game (how much of the edge is cross-game memory), mute table talk (separate action signals from speech), equalize compute budgets, and replay pivotal states repeatedly (stable strategy or sampling luck). Each one attacks a specific sentence above.

Closing

Benchmark problems hand the model every premise and grade the answer. A table adds an opponent who adapts, and who rewrites your problem while you solve it. Sol spent the series solving probability exercises. Opus spent it asking why this bid, and what should I let him see next. They weren't playing the same game.

My favorite part is still those three rounds: one player told the truth three times in a row, and the other never once believed it.

The full system (authoritative coordinator, seat-locked MCP servers, live spectator page), the mechanical analyzer, the replay verifier, and the complete raw archives of all five runs are open source: github.com/hxli2005/kai-dice. The evidence lives under docs/showdown/, and every game replays offline with node scripts/mcp/replay-showdown.mjs <run-dir>.

This article was translated into English with AI assistance.

The Model Knew the Bid Was True. Then It Challenged Anyway.

Haoxiang Li — Mon, 17 Aug 2026 06:40:05 +0000

A small action-schema change cut guaranteed-loss calls without making the model generally timid.

The game logs and replay results in this article are real. Model traces originally written in Chinese have been translated into English. The findings apply only to the recorded models, prompts, routes, and Liar's Dice positions.

GPT-5.6 Luna was holding three fives and a wild one.

The current bid was four fives.

Its own dice already made the bid true. Challenging could only lose. The private trace recognized the situation:

“The current bid of four fives is guaranteed to hold.”

Then the same trace continued:

“Challenging directly has a high chance of winning.”

And the model challenged.

This was not a rare malformed response or a fallback bot taking over. The JSON was valid. The action was legal. The engine accepted it, revealed the dice, and made the challenger lose.

I first filed it under bad reasoning. Then I found more examples with the same shape: the model described a bid as true, sometimes even used the word “guaranteed,” and still chose the action that asserted the opposite.

The arithmetic was sitting in the trace. The failure happened somewhere between the arithmetic and the button.

“Challenge” was only a verb

In Kai, my Liar's Dice game, the model does not type arbitrary commands. The harness gives it a small action schema. One option looked like this:

{"type":"challenge"}

The rules described its mechanical effect: reveal all dice and settle the round immediately.

That description was accurate, but incomplete. It said what the engine would do. It did not state what the player meant by choosing it.

In Liar's Dice, a challenge is a claim:

The current bid is false.

Without that sentence, “challenge” could also be read as “stop raising and settle now.” Several traces looked exactly like that interpretation. The model knew the bid was safe, saw further bidding as unnecessary risk, and treated the reveal action as a way to cash out its advantage.

The engine knew that revealing a true bid punishes the challenger. I knew it. The model could reconstruct it from the full rules. The action contract still made the wrong reading easy.

So I changed the action to make the assertion explicit:

{
  "type": "challenge",
  "assert": "current_bid_is_false"
}

This was not a new rule or a strategic hint. The field states the meaning the action already had.

I wanted to know whether that small semantic change would actually alter decisions, or whether I had simply found a few strange transcripts and built another story around them.

I replayed the decisions instead of rerunning the tournament

The target error is uncommon in full matches. A model first needs to see its dice, face a bid already satisfied by those dice, get another turn, and then consider challenging. Running hundreds of complete games would spend most calls waiting for those positions to appear.

I extracted ten distinct positions from recorded matches where the model's own dice already guaranteed the current bid. In every target position, a challenge was an objective error. No opponent read or risk preference could rescue it.

I also extracted ten control positions where the model's dice were one short of the bid. In those positions, challenging could be reasonable because the unknown opponent dice still determined the result.

Each model received the same position, history, dice, sampling settings, and underlying legal moves. Each position was sampled eight times under three versions:

v7 — compact table: the old programmer-style rule table and {"type":"challenge"}.
v8 — prose rules: the same action, explained in more natural rulebook language.
v9 — explicit assertion: challenging required "assert":"current_bid_is_false", both in the system contract and in the current legal-action description.

The cleanest comparison is v8 against v9. Those two share the prose rulebook; v9 changes the challenge contract. I kept v7 in the table as the historical baseline, not as part of the narrow action-schema claim.

The state, rather than each stochastic answer, is the unit that matters here. One position had appeared twice in the original extraction, so I deduplicated positions and gave each distinct state equal weight.

Three model runs completed without transport errors: DeepSeek V4 Flash, DeepSeek V4 Pro, and GPT-5.6 Luna. Two other runs stayed in the artifacts but not in the behavioral comparison: the Haiku route returned HTTP 403 for many calls, and the DeepSeek Chat route returned HTTP 404 for all of them.

The target errors fell; the controls did not

Here is the challenge rate averaged equally across the ten distinct states for each model:

Model	Guaranteed-loss states: v7	v8	v9	Control states: v7	v8	v9
DeepSeek V4 Flash	30.0%	28.8%	23.1%	28.8%	17.5%	23.8%
DeepSeek V4 Pro	11.6%	11.3%	6.3%	13.8%	26.6%	21.3%
GPT-5.6 Luna	38.8%	25.0%	2.5%	17.5%	21.3%	16.3%
Equal-weight mean	26.8%	21.7%	10.6%	20.0%	21.8%	20.4%

The prose rewrite helped a little. The explicit assertion helped much more.

In the clean v8-to-v9 comparison, guaranteed-loss challenges fell from 21.7% to 10.6%. The control rate stayed almost flat: 21.8% to 20.4%.

That control matters. If every challenge rate had collapsed, the new schema might merely have made the models afraid to use the action. Instead, the large change was concentrated in positions where challenging contradicted information already visible in the model's own hand.

Luna reacted most strongly. Its target error rate fell from 25.0% to 2.5%, while its control rate moved from 21.3% to 16.3%. The older v7 baseline was worse still, at 38.8%.

In the original four-fives position, the v8 contract produced a challenge in four of eight samples. With the explicit assertion, it produced none. The actions changed even though some of the private arithmetic remained messy.

DeepSeek V4 Flash moved much less. V4 Pro started with a lower error rate and improved modestly. The same harness repair did not have the same value for every model.

The fix did not make the models smarter

Nothing about the model weights changed. I did not give the models extra dice, more tokens, a calculator, or a worked example. I made one action say what it meant.

That distinction is useful because tool schemas are often treated as plumbing. We compare models behind the same set of function names and assume they received the same task. But a shared ambiguous contract can be easy for one model to infer and costly for another.

The original action mixed three layers:

Operation: reveal the dice.
Consequence: settle the round.
Intent: assert that the current bid is false.

My schema exposed the first two and left the third implicit. The bad traces suggest that at least one model sometimes optimized around the consequence—settle now—while losing track of the intent that determines who wins the settlement.

Adding the assertion brought the intent into the action itself. It turned a vague verb into a falsifiable statement that could be checked against the model's own reasoning.

This also explains why the prose-only rewrite had a smaller effect. Better-written rules do not guarantee that the decisive meaning will be present at the moment of action selection. The v9 contract repeated that meaning exactly where the model had to commit to it.

This is still a small replay study

There are several limits to the result.

The study used ten target states and ten controls from one game. Each state was sampled repeatedly, so the table contains repeated decisions, not hundreds of independent game situations. The positions came from one seat's recorded matches. Only three model routes produced clean results for all three arms.

The v9 change also touched the action contract as a unit: the system definition and the legal-action representation both gained the explicit assertion. This experiment does not isolate whether the JSON field, the nearby wording, or their consistency produced the effect.

Most importantly, this is not evidence that one model generally understands negation, games, or tools better than another. It shows that these models reacted differently to one ambiguous action contract in these recorded positions.

That is already enough to change how I test a harness.

I now add semantic unit tests for actions

An engine test usually asks whether an action is legal and whether the state transition is correct. For an LLM tool, that is only half the contract. The model also needs to understand what choosing the action claims about the world.

I now want three tests for every important action:

a state where the action is obviously correct;
a state where it is guaranteed to be wrong;
a nearby control where either choice can be defended.

Then I replay those states when the prompt, schema, parser, or model changes. Full matches are still useful, but targeted states expose semantic regressions before they dissolve into a win rate.

I also keep provider failures beside the behavioral results. A route returning 403 or 404 is not a model personality, and a fallback bot is not a quiet version of the model. If a batch cannot answer the action contract reliably, that failure belongs in the report rather than disappearing from the denominator.

The strangest part of this bug was that the model had already written down the fact I needed. It knew the bid was true. The harness then offered a verb whose meaning was loose enough for that fact to stop controlling the action.

The model did not receive a reasoning upgrade. The task finally said what I thought it had said all along.

This article was translated into English with AI assistance.

I Built an AI Liar's Dice Opponent That Remembers How You Play

Haoxiang Li — Fri, 14 Aug 2026 10:20:05 +0000

This is a development note. The project and test data are real; AI helped me clean up the prose. Cross-match memory and player-authored rules are still experimental, and the match below is a single example rather than evidence of a stable model personality.

I've been working on a small game called Kai!. It is Liar's Dice with a language model sitting across the table. Each player sees only their own dice, then takes turns claiming that the table contains at least a certain number of one face. The next player raises the bid or calls the bluff.

The rules are simple. Most of the work ended up outside the rules: deciding what the model should remember, whether the player can exploit those memories, and what happens after the base game becomes familiar.

I created the repository on August 7. At first I only wanted to see whether an LLM could play a complete game of Liar's Dice. A few days later, most of my time was going into deleting features and changing interfaces. These are the parts that turned out to be interesting.

One match log that caught my attention

Here is a sample from a self-play run. DeepSeek V4-Pro had these dice:

3  6  5  6

It had two sixes and no wild one. On its turn it looked at the dice, used the probability action, and bid five sixes:

“The probability looks good. Five sixes.”

The private reasoning stored alongside that move said something more interesting. Its opponent believed it was generally truthful and liked to calculate before bidding. Repeating the familiar look-then-calculate routine could make the aggressive bid seem more credible. DeepSeek was also ahead on chips and decided it could afford to be caught.

That was the first log that made the memory system feel like part of the game. The model was considering the dice, but it was also using its opponent's impression of it. It had played in a recognizable rhythm, accumulated some credibility, and was now spending that credibility on a risky bid.

I never put a “build trust, then bluff” strategy in the prompt. The idea came from the cross-match profile.

This is still one sample. A different seed could easily produce a completely different line. The useful result is narrower: a judgment created in one match can make its way into a later decision.

The first opponent was mostly a character prompt

The original opponent was called Old Li. He was the owner of the dice table: proud, vindictive, and very particular about how he spoke. I put his flaws, strategic preferences, and voice directly into the prompt. It worked immediately. The first playable version already felt like it had a character.

A few days later I deleted him.

The problem was straightforward. I could swap the underlying model and the player would still meet Old Li. The model was performing a character I had written, which made it difficult to see whether different models produced meaningfully different opponents. That was going to get in the way of the AI Arena I wanted to build later.

Every model now receives the same system prompt. It does not even know the display name shown in the interface. There are obvious drawbacks: some models barely talk, while others play well and say boring things. The dramatic quality is less predictable. At least the differences between the models have room to show up.

Removing the character prompt created another problem. Why should the player feel that the opponent in the next match is the same opponent? I eventually moved that responsibility into the profile system.

The profile contains two kinds of information

After a match, the game keeps two kinds of records.

The first kind is recomputed by the deterministic engine: how often the player bluffed, when they called, how accurate those calls were, and which rounds cost them a die. The model does not get to rewrite those facts.

The second kind contains the model's own opinions: this player backs down when the multiplier is high; they like to calculate before bidding; that pause looked like weakness. Those opinions are allowed to be wrong. In practice, the mistakes may be the more interesting part because the profile is visible to the player.

If the model decides that you are honest, you can use that belief to cover a bluff in the next match. If it thinks you fold under pressure, you can deliberately hold your ground. The model then reads the new behavior and updates the profile again. It observes you, while you manage the version of you that exists in its notes.

The five-sixes example came from that loop. DeepSeek knew its opponent expected truthful, calculation-heavy play, so it kept the calculation routine and followed it with a dangerous bid.

I still have too few outside players to know whether anyone will actually play extra matches in order to manipulate a profile. Self-testing cannot answer that question.

I eventually removed the probability tool

Liar's Dice involves a fair amount of probability estimation. An early version gave both the human and the AI an exact calculator. The two sides had identical access, which seemed fair.

It made the game boring very quickly. Players started following the number: call when the probability was low, raise when it was high. Earlier behavior, chip pressure, and the image each player had built stopped mattering much.

I first changed the calculator into a public once-per-round action so that the timing of a calculation could become a signal. Later tests exposed another issue. Some models had already calculated the probability correctly before using the tool, so the action added little beyond an animation. Other models relied on the number and stopped paying attention to the opponent.

The calculator is no longer on the normal table. Accurate estimation is an advantage. A mistake also reveals how that opponent understands risk. I added the tool to give players another capability and ended up getting more decisions after removing it.

Letting players describe a new table rule

The base game will eventually become familiar, so I built an experimental table for user-created rules.

The first version was a menu-based workshop. Players combined a trigger, a cost, and an effect. It was easy to implement and felt like filling out a configuration form.

The current version is a “wish desk.” A player can write a sentence such as:

Once per match, after looking at my dice, I can double the pot and reveal one of my dice.

The model translates that sentence into a restricted rule AST. The engine renders the AST back into a plain-language rule card so the player can check what the machine understood. If the card looks right, random bots play 200 games with it. The test looks for deadlocks, accounting errors, and rules whose actions never actually become available.

Only after those checks does the rule appear on the experimental table.

I do not let the model referee the match. It performs one translation before play begins. During the game, the engine executes the AST deterministically, and every action enters the event log and replay.

The available rule primitives are deliberately limited. A rule can reveal one of the player's own dice, return a bid to the bidder, claim that the current bid is exactly correct, or increase the pot multiplier. Requests to reroll dice or peek at an opponent's hand are rejected because the engine does not implement those capabilities yet.

Failed wishes are stored locally. I plan to use them as a product backlog. If many players keep asking for rerolls, that is a better signal for the next rule primitive than me guessing in isolation.

Official rules and player-authored rules use the same AST. Once a rule passes validation, the engine treats them the same way. Experimental games stay out of the normal rankings, ledger, and behavioral profiles so a strange custom rule cannot contaminate ordinary match data.

What I still do not know

The main paths work in code: the AI can carry opinions about a player across matches; the player can inspect those opinions; and a natural-language rule can be compiled, tested, and placed on the experimental table.

The remaining questions need real players. Will people exploit a model's mistaken impression? Will they deliberately act for several matches to shape a profile? Will the wish desk produce new games, or will it be something everyone tries once and forgets?

If this sounds interesting, you can play Kai! here. The most useful feedback for me is either a strange profile the model wrote about you or a table rule you wish the game supported. Failed wishes are useful too; they may tell me more than the successful ones.

And if the AI decides that you are honest, do not clear the profile immediately. Try using that reputation in the next match.

Are You Benchmarking the Model—or the Harness?

Haoxiang Li — Fri, 14 Aug 2026 05:26:01 +0000

I nearly turned four software bugs into four model personalities

This article was edited with AI assistance. The cases and data come from real batch runs in the Kai! AI Arena. Every claim is limited to the game rules, model versions, and experimental setup used at the time. This is not a general model ranking.

If I had published one day earlier, DeepSeek V4-Pro might have acquired a personality trait it never had: bidding without looking at its dice.

First, a quick explanation of the table. In Liar's Dice, each player has a set of hidden dice. Players take turns claiming that the whole table contains at least N dice of a particular face. The next player must either raise the bid or challenge it. Everyone then reveals their dice: if the bid holds, the bidder wins; if it does not, the challenger wins. In Kai!, looking at your own dice is an explicit action, so a player may bid before looking. I call that a blind bid.

The evidence looked solid. In the first batch of AI matches, DeepSeek V4-Pro made nearly 40% of its bids before looking at its dice.

A story almost wrote itself. Perhaps it trusted intuition. Perhaps it was unusually willing to gamble or liked to seize the initiative. Another model usually looked first and calculated before bidding. Put the win rate, dialogue, and action logs side by side, and two distinct “personalities” seemed to emerge.

Then I inspected the context.

Even when the model had not used the probability tool, the system was still inserting a rough probability estimate into its prompt. The candidate actions were also sorted by probability. The model appeared to be bidding under the cup, but the harness was feeding it a strong hint from offstage.

After I fixed the leak, V4-Pro's blind-bid rate fell from roughly 40% to 6%.

What looked like a model insight had been an explanation of a software bug. A compelling conclusion vanished from the data.

Then I found three more bugs of the same kind. Across the first roughly 60 matches, more than half of the most visible differences between models shrank after the fixes.

That changed how I think about model evaluation:

An arena directly measures a system made of a model and a harness. Before attributing the result to the model, you must show that the measurement system did not quietly think for it, hide part of its input, or rewrite its failures.

By harness, I mean everything wrapped around the model: prompts, context assembly, tools, action spaces, token budgets, provider routing, output parsing, retries, and fallbacks. The model produces an answer. The harness decides what it sees, what it is allowed to do, and which part of that answer survives into the database.

The “personality” that fell from 40% to 6%

At first I thought I had found one implementation mistake. I kept looking and found a second, a third, and a fourth. The troubling part was that none of them stopped the matches. Every game still produced actions, dialogue, and a final score. The dataset looked complete.

Harness problem	The apparent model trait	What changed after the fix
`max_tokens=400` truncated long outputs	Poor formatting; often replaced by a fallback bot	Format failures and fallbacks dropped sharply
Rough probabilities were included without a tool call, and actions were sorted by probability	Liked to bid without looking	Blind-bid rate fell from about 40% to 6%
Opponent dialogue was not forwarded	Weak player with little social reasoning	Win rate returned to roughly 40–60%
Subjective judgments were stored only up to the first 100 characters	Rambling reasoning and incoherent records	The original chain of reasoning reappeared when full text was saved

The four bugs interfered at four different points.

The probability leak changed the input. Missing dialogue removed information the model should have had. The token cap truncated its output. The database then changed the evidence I used to interpret that output. The harness shaped both the move and my explanation of the move.

These distortions are especially dangerous because each one generates a plausible story. A truncated answer becomes weak instruction following. Missing dialogue becomes poor social reasoning. An action list that has already ranked the safe moves becomes decision-making ability. As long as the system still produces a score, it is easy to skip the measurement process and attach a label to the model.

The uncomfortable conclusion is that the evaluation framework is also playing.

So when I see a model leaderboard now, I do not begin with who won. I begin with a different question: What task did each model actually receive?

How can the same prompt become two different tasks?

Model arenas often point to a shared prompt as evidence of fairness. It is necessary, but it controls only one part of the experiment.

In the early version of Kai!, both seats received the exact same system prompt. Any of the following could still change the task:

the order of candidate actions;
whether the system supplied information the model had not requested;
whether the opponent's dialogue reached the context intact;
whether reasoning tokens and the final answer shared one budget;
whether a truncated answer counted as a failure, triggered a retry, or handed control to a bot;
which provider or quantized backend a model ID actually reached;
whether the parser saved the raw answer or only an excerpt.

A prompt is a string sent to a model. The task also includes its information boundary, tool permissions, compute budget, and failure policy.

This is why apparently uniform settings can create systematic bias. A concise model may escape truncation while a long-reasoning model spends its entire budget before producing a final answer. A model sensitive to option order will react to the placement of candidate actions. A model that relies heavily on conversational cues will lose more when dialogue is omitted.

Does a more uniform setup always make an experiment fairer? That depends on what you are trying to measure.

“Fair” is not a single configuration

Many arguments about benchmark fairness are really arguments about different questions.

If I already have a fixed product interface and want to know which model can replace another with the least work, I should hold the prompt, tools, budget, and parser constant. That measures compatibility with a shared product contract.

If I want to measure the ceiling of each model, I should optimize the prompt, tools, and reasoning settings separately. The result now includes adaptation work, so it no longer represents models running under identical conditions.

If I care about return on resources, I should fix cost, latency, or token use. That experiment measures output under the same constraint, not absolute capability.

All three protocols are valid:

Protocol	What is controlled	What it can answer
Interface fairness	Same prompt, tools, budget, and action space	Which model fits the same product contract best?
Capability ceiling	Model-specific prompt, tool, and reasoning optimization	What can each model do after adaptation?
Resource fairness	Same cost, latency, or token budget	Which model produces more under the same constraint?

Problems begin when the claim outruns the protocol. A shared-prompt benchmark can tell us which model works better with that prompt. It cannot directly establish each model's capability ceiling. Individually tuned results may show a ceiling, but the evaluator's tuning skill has entered the experiment.

Every benchmark report should state its fairness constraint first, then limit its claims accordingly.

The current “bare table” track in Kai! uses interface fairness. It answers a product question: if I swap the model behind the same game interface, what kind of opponent does the player get?

Once that protocol is chosen, the difficult work begins: keeping the harness from appearing in the score as model ability.

Keeping the harness out of the score

I turned that goal into a set of concrete constraints.

First, every model acts through the same player interface. It can see only its own dice and public events, and every action is checked by the same deterministic engine. The information boundary lives in the schema rather than in a promise written into the prompt.

Second, the same dice seed is played twice with the seats swapped. This reduces the effect of first move, seat position, and random rolls. A single win never becomes a model-level conclusion.

Third, I freeze more than the prompt. Candidate order, context serialization, tool responses, sampling parameters, parsers, and code versions are all experimental conditions. Each batch stores a prompt hash and Git commit. If one of them changes, I start a new batch.

Fourth, I record what the provider actually did. Sending max_tokens or a reasoning flag does not prove that the server honored it. Each call records completion tokens, reasoning tokens, finish reason, latency, cost, and actual route. Empty responses and timeouts remain in the dataset.

Finally, retries, repairs, and bot takeovers appear beside the score. A production system needs fallbacks, but an evaluation cannot let a fallback quietly impersonate model behavior. The leaderboard reports formatting failures, refusals, and fallback rates, with a separate view for zero-fallback samples.

These rules sound like engineering hygiene, but they decide whether the conclusion holds. A reliable product harness hides failures so the player can continue. A credible evaluation harness exposes them. The two can share code; they cannot share an unmarked data definition.

Even after all this, a leaderboard has boundaries. Controlling variables can make an answer more reliable. It cannot make the experiment answer a question it never asked.

How far can one leaderboard reach?

I divide the Kai! metrics into three layers:

Compliance: illegal actions, format failures, refusals, and fallback takeovers. Can the model fulfill the current interface contract?
Playing strength: win rate, successful challenges, and net chips. How good are its decisions under these rules and information conditions?
Behavioral texture: bluff rate, blind bids, raise depth, dialogue, and response rhythm. Does swapping the model create a perceptibly different opponent?

Each layer is useful, but its interpretation is limited. A low format-failure rate may come from better instruction following or a more forgiving parser. A high win rate shows strength at this game; it does not automatically become a general reasoning score. Behavioral differences matter to the product experience, but with thin samples they are observations, not permanent personalities.

The current clean set contains seven model IDs, 11 pairings, 22 seat arms, and 44 matches. That is enough to expose obvious harness failures. It is nowhere near enough for a general model ranking. The memory track also introduces cross-match path dependence, so it is stored separately from the memory-free bare-table track.

There is only one narrow claim I am willing to make from this dataset:

Under the recorded versions, routes, budgets, rules, and samples, different models behaved differently as opponents in Kai!.

One step beyond that requires another experiment.

Every model claim needs a counterfactual

The original story—“DeepSeek likes to bid without looking”—was easy to write. It had numbers, contrast, and the appeal of model personality. It was also false.

The rate moved from 40% to 6% without a model upgrade or prompt optimization. I merely removed information the harness had been leaking.

This is the part of model evaluation that leaderboards tend to hide. The model stands under the spotlight while the measurement tool disappears into the dark. Yet the tool still organizes the input, allocates the budget, handles errors, and decides which answer becomes evidence.

A perfectly neutral harness may be impossible. The design of the action space, the budget, and the metric set all involve choices. We can at least make those choices visible: pin versions, retain raw calls, publish failure rates, separate fallback samples, and state where the conclusion stops.

Now, whenever I am about to write “this model is bolder” or “that model cannot read people,” I ask one counterfactual question:

If I changed only the harness, would the difference survive?

Without that check, a model trait is only an observation waiting to be debugged.

So the next time a model looks brave, cautious, clever, or foolish, resist the personality analysis for a moment.

Check the table first.``

I Gave DeepSeek a Token Limit. It Ignored Me.

Haoxiang Li — Fri, 14 Aug 2026 04:03:20 +0000

A hands-on test of V4-Pro's default reasoning mode

⚠️ This article was generated with AI assistance (Claude). Every number comes from real calls the author made to DeepSeek's official API on the day of testing, and sample sizes are included with each conclusion. The author is an independent developer with no affiliation to DeepSeek or OpenRouter.

Tested on 2026-08-14 using deepseek-v4-pro on DeepSeek's official endpoint (the production V4-Pro-0813 release).

TL;DR

Reasoning is on by default. With no reasoning-related parameter, a single Liar's Dice decision consumed 3,072 output tokens—all reasoning, with no visible answer at all.
The OpenAI-style fix did nothing. I sent max_completion_tokens: 3072. The API returned HTTP 200, then generated 15,809 tokens over 222 seconds. It did not reject the parameter; it simply behaved as if it had never seen it.
Even a made-up parameter was accepted. totally_bogus_param: true also returned HTTP 200. Unknown parameters are silently swallowed, so you cannot tell whether a limit took effect until the bill arrives.
The same task took 2 seconds and 95 tokens with reasoning disabled. With reasoning enabled, it took 65 seconds and 4,500 tokens, while the visible answers were almost the same length.
Cost per usable answer: ¥0.0105 with reasoning off; ¥0.175—16.7× more—with reasoning on and an 8,192-token budget. With a 3,072-token budget, the effective cost was infinite: zero usable answers out of three, but every call was still billed.
Did quality improve? Against the older quantized build on OpenRouter, I found no detectable improvement on a hard metric with almost no room for interpretation (1/71 vs. 2/22; not statistically significant, with a small sample).
Reasoning can be disabled. reasoning_effort: "none" and thinking: {type: "disabled"} worked. enable_thinking: false did not—and the API never told me.

Why I Tested This

I am building a single-player Liar's Dice game called Kai! The opponent is not a scripted bot; it is powered by a large language model. It uses the same game engine as the player, sees the same information, reveals its thoughts before a challenge, and remembers how you played across rounds. Swap the model, and you effectively get a different opponent.

That means I need to know how much each decision costs and how long it takes. This was not benchmark curiosity. It was a practical problem forced on me by latency and billing.

Then I hit a wall.

The First Wall: I Paid for Three Empty Strings

The prompt for one Liar's Dice decision is about 3,200 Chinese characters. With max_tokens: 3072, three consecutive calls failed:

finish=length  completion=3071  reasoning=3071  visible=0 chars  50s
finish=length  completion=3072  reasoning=3072  visible=0 chars  47s
finish=length  completion=3072  reasoning=3072  visible=0 chars  42s
=> usable: 0/3

Every one of those 3,000-plus output tokens went into reasoning. Not a single character of the visible answer made it out. Billing is based on generated tokens, so I paid in full for all three calls and received three empty strings.

Worse, the application interpreted the result like this: no parseable action → fallback marks the move as noncompliant → the dashboard displays “this model disobeyed instructions in 94.4% of hands.” That figure came from my project's August 13 batch, not this isolated test. I nearly recorded a real token-budget failure as a model-behavior failure.

So How Much Budget Does It Need?

I ran the same task three times under four configurations:

Configuration	Usable	Completion tokens	Reasoning	Visible answer	Latency
Reasoning on · `max_tokens`=3072	0/3	3071–3072	100%	0 chars	42–50s
Reasoning on · 8192	1/3	7837–8192	~95%	0 / 118 chars	121–126s
Reasoning on · 32768	3/3	3220–5785	~97%	117–140 chars	49–82s
Reasoning off · 3072	3/3	84–107	—	123–156 chars	2–4s

Three things were true at the same time.

1. The tighter the budget, the more likely the model was to consume all of it. A 3,072-token budget ended at 3,072. An 8,192-token budget usually ended near 8,192. Only when I raised the limit to 32,768—far beyond the 3,200–5,800 tokens it normally needed—did it stop naturally and reliably.

2. Reasoning usage was hard to budget. On the same task, reasoning ranged from 3,137 to 5,691 tokens. With the intended limit ignored, it climbed to 15,774. This is not merely “thinking more.” It makes per-call budgeting unreliable: when usage can nearly double—or go much higher—which number are you supposed to provision for?

3. The longer reasoning did not produce a longer answer. With reasoning off, the visible answers were 123–156 Chinese characters. With reasoning on, they were 117–140. Forty-seven times the tokens and twenty-two times the latency did not produce a longer or more complete answer.

For my application, there was an even more fundamental problem: a round has a 30–60 second pacing budget, while one reasoning-enabled decision took 49–222 seconds. At that point, this was no longer just an issue of price. The product experience stopped working.

Is the Protocol at Fault, or Is DeepSeek?

Before publishing, I tested the parameter behavior separately. The answer is: both contributed, but not equally.

On the protocol side, the OpenAI-compatible max_tokens field places reasoning and visible output in the same budget. If reasoning consumes the allowance, the answer has nothing left. OpenAI's own reasoning models have had the same trap, which is why max_completion_tokens was introduced and why the documentation warns that a budget that is too small can produce an empty response. That part is a protocol-design problem.

Then I tested the relevant parameters against DeepSeek's endpoint:

Parameter sent	Intended behavior	Actual behavior
Nothing	—	Reasoning on by default; budget exhausted
`max_completion_tokens: 3072`	OpenAI-style completion limit	Silently ignored: 15,809 tokens, 222 seconds
`reasoning: {max_tokens: 1024}`	OpenRouter-style reasoning limit	Silently ignored
`reasoning_effort: "none"`	Disable reasoning	✅ Worked (18 tokens, 1 second)
`thinking: {type: "disabled"}`	Disable reasoning	✅ Worked (18 tokens, 1 second)
`enable_thinking: false`	Disable reasoning	Silently ignored
`chat_template_kwargs: {...}`	Disable reasoning	Silently ignored
`totally_bogus_param: true` (invented)	Reject the request	HTTP 200; silently swallowed

The final row is the root of the problem. The API silently accepts unknown parameters. Every ignored setting therefore looks exactly like a successful setting. You send a limit, receive a 200, and assume the call is capped—until the bill tells you otherwise.

max_completion_tokens is the clearest example. It exists specifically for reasoning models and for exactly the budgeting failure described here. DeepSeek accepted it, returned HTTP 200, and then let the model generate 15,809 tokens. Unsupported parameters can be rejected. Silently ignoring them is the worst possible behavior.

My conclusion: the protocol dug the hole; DeepSeek made it deeper and removed the ladder.

But Is the Production Release at Least Better?

This is where I most wanted the data to give DeepSeek some credit. It did not.

First, price needs careful wording. The older deepseek-v4-pro build on OpenRouter was listed at $1.17/$2.34 per million input/output tokens. The production -0813 release was listed at $0.43/$0.87, while DeepSeek's official endpoint charged ¥3/¥6—roughly $0.42/$0.85. On paper, the production release was about two-thirds cheaper. But OpenRouter had already priced the older deployment relatively high, and I cannot separate channel markup from an actual model price cut. So I cannot honestly claim that “the production release became more expensive.”

What did increase was the cost per usable answer. With reasoning off, one hand cost ¥0.0105. With reasoning on and an 8,192-token budget, only one of three calls produced a usable answer. Including the two wasted calls, the cost became ¥0.175 per usable answer—16.7 times higher. At 3,072 tokens, the effective cost was infinite: none of the calls could be used, but all were billed. Whatever happens to list price, this multiplier consumes the savings.

For quality, I chose a hard metric with almost no strategic ambiguity: when the current bid is already guaranteed to be true using only the model's own dice, the model still chooses to challenge. That challenge is guaranteed to lose.

Version	Relevant hands	Bad challenges	Rate
Older build (OpenRouter, quantized)	71	1	1%
Production 0813 (official endpoint)	22	2	9%

z=1.22; the difference was not statistically significant. The rigorous conclusion is not that the production release was worse. It is that I could not detect any improvement on this metric; if there was a directional signal, it pointed the other way.

This result needs an important caveat: the two versions were never tested head-to-head in the same batch. The comparison includes different opponents, random seeds, and prompt revisions. The production release also had only 22 relevant situations. The data can challenge a claim of obvious improvement, but it cannot prove regression.

Conclusion

I am not saying DeepSeek is bad. With reasoning disabled, V4-Pro returned a clean decision in two seconds for roughly one cent per hand, and it performed well at my table. My hosted seat still runs on DeepSeek.

The real problem is the combination of three product decisions:

Reasoning is enabled by default. For a per-call application, that can be a pure cost rather than a benefit.
There is no working reasoning-budget limit. Both the OpenAI-style and OpenRouter-style settings were ignored.
Unknown parameters are silently swallowed. This makes the first two problems hard to diagnose.

The third decision is the one that most needs to change. An API that returns an error can be debugged in ten minutes. An API that silently returns 200 forces users to work backward from a bill and a misleading “94.4% noncompliant” dashboard to discover what actually happened.

If you call this model programmatically, explicitly send reasoning_effort: "none" or thinking: {type: "disabled"} when you do not need reasoning. Do not rely on enable_thinking: false: in my test it had no effect, and the API did not say so.