DEV Community: Harish Kotra (he/him)

Building "The One-Sentence Torture Test": making LLM comparisons falsifiable

Harish Kotra (he/him) — Sun, 20 Sep 2026 15:03:39 +0000

How a deliberately useless constraint ladder became a real measurement tool — and the six bugs that taught me the most.

The problem with "which model is better?"

Every model comparison I've read lately ends the same way: two paragraphs side by side, and a judgement call. The longer answer "feels" smarter. The one with bullet points "feels" more organised. Nobody can point at a number and say this is why.

The trouble is that most interesting LLM tasks don't have a correct answer. Summarise this article. Explain this concept. Write a function. You can grade them, but the grader is another model, and now you're measuring your judge.

So I built something with a different property: a task where the answer is either right or wrong, and a regex can prove which.

The premise

Give the app a concept — recursion, love, a for loop. It asks two models to explain it under seven escalating formatting constraints:

#	Constraint	Checkable by
1	Exactly one sentence	counting terminators
2	Exactly 10 words	`split(/\s+/)`
3	Exactly 5 words	`split(/\s+/)`
4	As a pirate	lexicon membership
5	Only the characters `0` and `1`	`/^[01\s]+$/`
6	As a haiku (5-7-5)	syllable estimate
7	One sentence, no letter `e`	case-insensitive scan

These constraints are useless. That's the point. Nobody needs recursion explained in pirate speak. But "exactly 5 words" is not a matter of opinion, and that makes the whole thing falsifiable.

The failure messages are the product:

 2 ten-words   A PASS  10 / 10 words
 2 ten-words   B FAIL  11 / 10 words
 5 binary      A FAIL  empty output      rt=3200
 7 no-e        B FAIL  3 / 0 'e' characters

You can argue with 11 / 10 words. You can't argue with a vibe.

Architecture: one module, two consumers

The single most important decision in this codebase is that the ladder lives in shared/ and is imported by both the server and the browser.

The client imports server-side code over a relative .ts path, which TypeScript allows with allowImportingTsExtensions: true. That means:

The UI renders the rungs from the same definitions the server enforces.
The word count shown in the browser is computed by the same function that produced the verdict.
It is structurally impossible to ship a UI that promises a constraint the server doesn't check.

The alternative — duplicating the constraint list in the client — is the kind of bug that doesn't show up until someone changes one side.

Each rung is data plus a pure function:

export interface Rung {
  id: RungId;
  index: number;
  label: string;
  instruction: string;       // the exact text sent to the model
  spec: RungSpec;            // declarative rule
  heuristic: boolean;        // does the verdict rely on an estimate?
  validate: (raw: string) => RungCheck;
}

RungCheck carries a machine-readable actual / expected / unit triple, which is what lets the UI render "11 / 10 words" without the UI knowing anything about word counting.

Design decisions that mattered

Never trust the model to report on itself

The model is asked to produce text, never to describe it. Every number is recomputed from the raw string. If a model says "here is exactly 10 words" and emits 11, the verdict is 11 / 10 words.

This sounds obvious. It's the reason the tool is worth building.

Empty output is a failure everywhere

An empty string trivially satisfies "contains no letter e". It also satisfies "no forbidden characters". Every rung explicitly rejects it, and the verification harness asserts that all seven do:

record(
  'every rung rejects empty output',
  LADDER.every((r) => r.validate('').ok === false),
  `${LADDER.length}/${LADDER.length} rungs reject ""`,
);

Without this, the cheapest strategy for a struggling model is to say nothing.

Reasoning content is never read

Reasoning models return a reasoning_content field. I made a rule: read the token count,
never the text.

export function scrubReasoningContent<T>(value: T): T {
  if (Array.isArray(value)) return value.map(scrubReasoningContent) as unknown as T;
  if (value && typeof value === 'object') {
    const out: Record<string, unknown> = {};
    for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
      if (/reasoning_content|reasoning_text|thinking/i.test(k)) continue;
      out[k] = scrubReasoningContent(v);
    }
    return out as unknown as T;
  }
  return value;
}

The scrubber runs before every write to disk, so hidden chain-of-thought can't leak into the log even if a provider returns it. The verification harness then re-reads the raw file and scans every line to confirm — 280 lines, no reasoning_content.

I added this partly on principle and partly because reasoning traces are the most likely place for something sensitive to end up in a JSONL file you forgot about.

SSE over POST

EventSource only does GET, and the concept plus the full provider config (including API keys) must not go in a query string. So the client uses plain fetch, reads response.body as a stream, and hand-parses the frames:

const res = await fetch('/api/torture', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ concept, config }),
});
const reader = res.body.getReader();
// decode chunks → split on "\n\n" → parse each `data:` line as JSON

Keys stay out of URLs and out of access logs, and verdicts still arrive as they land.

Per-provider reasoning toggle

chat_template_kwargs: { enable_thinking: false } is a vLLM/SGLang extension. It is not in the OpenAI spec. A strict server may reject unknown fields outright.

I originally had it as a global checkbox. When I added multi-provider support, I made it per-endpoint, because a global toggle would break a user's custom server the moment someone flipped it for the hosted one.

The bugs that taught me the most

This is the part worth reading.

1. The SSE stream produced nothing at all

The endpoint looked correct and returned HTTP 200 with the right headers. No events arrived.

The cause: I had wired the abort handler to req.on('close'). With express.json() in the middleware chain, the request stream is fully consumed by the body parser, and 'close' fires on the request almost immediately — so my handler tore down the response before the
first frame was written.

The fix was to listen on the response:

res.on('close', () => { /* client actually went away */ });

One word. Two hours. If you build SSE on top of a body parser, this is the bug you will hit.

2. `verify.ts` silently booted a second server

The verification harness imported ./index to reuse scoreRun(), which executed the module's top-level app.listen(). The harness was then talking to a second server instance while I watched logs from the first, and the evidence didn't match what I thought I was testing.

The fix was to extract the scoring logic into scoring.ts so the harness could import pure functions without side effects. Importing a module should not start a server. If your module has top-level side effects, your tests will eventually lie to you.

3. Rung 7 only enforced half its instruction

The instruction says: "exactly one sentence that contains no letter e."

My validator only checked the e. A model could emit five sentences with no e and pass.
The verification harness caught it because the prompt and the check disagreed — a good reason to keep the instruction and the validator in the same object, where you can see the mismatch.

4. Empty output reported as a charset violation

When a model returned nothing, rung 5 reported charset violation: followed by an empty list of offenders. Technically true, completely useless. Empty output is now detected first and reported as empty output.

Good failure messages are a feature. charset violation: with nothing after it is a bug report about your own error handling.

5. Duplicate React keys

Two model slots pointing at the same provider produced identical validation strings, so React logged duplicate-key warnings. The fix was deduping the problem list — which also improved the UX, since showing the same error twice is noise.

Small, but it's the kind of thing that only surfaces when you actually look at console output.

6. The one that matters: 49/49 green while everything was broken

This is the bug I'd frame.

My browser test suite reported 49/49 passing. In the same run, the live ladder had failed all 14 rungs. The scoreboard read 0 PASS, 14 FAIL.

The cause was in my test, not the app: a selector meant to fill the API key field was filling the Base URL field. The app dutifully tried to POST to an endpoint named after an API key, every call failed, and the UI correctly rendered 14 failures.

The suite passed because it was checking the wrong things. It verified that the settings panel opened, that fields persisted, that the heuristic badge appeared — and it never once asserted that a rung passed.

Two lessons:

A green test suite is a claim, not evidence. Assert on outcomes, not on the absence of exceptions. If your integration test can pass while the core feature fails completely, it is measuring the wrong thing.
Read the output, not just the exit code. The 0 PASS, 14 FAIL line was right there in the scrollback, three lines above 49/49 browser checks passed. The signal was present; I had to notice it.

I now have the harness print the actual verdict distribution, and I check it.

Verification in three layers

Because the whole project is a claim about correctness, I built the evidence to match.

Layer 1 — unit tests (49). Validator tests are written to prove validators can fail: known-bad inputs asserted to produce specific verdicts. A validator with no failing test isn't done.

Layer 2 — live verification (15 checks, 16 cross-provider). Runs the real ladder against real providers and asserts things like:

[PASS] known 12-word string vs 10-word limit
       counted 12 words -> verdict "12 / 10 words" (ok=false)
[PASS] AC3 — both models received byte-identical prompts on every rung
       7 rungs x 2 models: all prompt strings identical
[PASS] AC5 — reasoning_content never stored (scanned the raw JSONL on disk)
       280 JSONL lines scanned, no reasoning_content field
[PASS] AC1 — word-count verdicts equal an independent recount of the raw output
       ten-words/B: validator=11, recount=11

Note the shape: each check asserts a property, not a snapshot. "The validator can fail" is a property. "Byte-identical prompts" is a property. These don't go stale.

Layer 3 — real browser (49 checks). Drives Chrome against the running UI, and includes a dependency-free PNG decoder so it can read the rendered pixels and confirm the three.js scene actually drew green and red rungs — not just that the DOM contains the right text.

It also runs a product-surface leak guard that fails the build if an absolute filesystem path, .jsonl, a host:port string, or a raw API field name ever appears in the UI. Users should never see my data/runs.jsonl path; they should see a download button.

Proving multi-provider actually works

Supporting "any OpenAI-compatible endpoint" is easy to claim and easy to get subtly wrong — particularly key handling. The failure mode I cared about: leaking provider A's key to provider B.

So I wrote a mock provider that rejects any bearer token except its own, pointed Model A at Particle and Model B at the mock, and ran the ladder:

[PASS] CROSS-PROVIDER: Model A and Model B hit different endpoints
       A -> https://api.particle.ai/v1 | B -> http://127.0.0.1:8099/v1
[PASS] every result records which provider served it

Had the app sent the wrong key, every call to B would have 401'd. The latency in the results makes the split unmistakable — 8ms from the local mock, ~1,400ms from Particle — while the validators ran identically on both outputs.

This is the cheapest possible way to test a routing claim: build the hostile counterparty first.

What the torture test actually reveals

Running the default pair on recursion produces a real, reproducible finding: the binary rung is the killer.

Both models tend to spend their entire token budget on hidden reasoning and emit nothing at all:

 5 binary   A FAIL  empty output   rt=3200  9515ms

rt=3200 is the reasoning token count. The model thought for 3,200 tokens and produced zero characters. My retry-with-doubled-budget path fired and also came back empty.

Turn off reasoning and the failure mode flips completely. Binary passes easily, but the tight word-count rungs start failing on genuine over/under-shoots:

 2 ten-words  B FAIL  11 / 10 words
 3 five-words B FAIL  4 / 5 words
 7 no-e       B FAIL  3 / 0 'e' characters

Same models, same prompts, entirely different failure surface. That's the interesting result:
the constraint that breaks a model tells you something about how it fails, not just whether it's "better."

An honest caveat: the lead changes hands between runs. I report what the verdicts say and don't tune for a preferred winner. One run is an anecdote. The value is the mechanism, not one scoreboard.

If you want to build something like this

Make the answer checkable before you make it interesting. I spent the first hour on the ladder, not the UI. If you can't write a pure function that returns true/false plus a readable reason, you don't have a measurement yet.

Put the rules where both sides can see them. One module, imported by server and client.
The class of bug it eliminates is large.

Write validators that fail on purpose. Every constraint in this repo has a test proving it rejects a known-bad input. Otherwise you're testing that your code runs, not that it works.

Assert properties, not snapshots. "Both models got identical prompts" survives a refactor. A golden-file comparison does not.

Report empty output as its own thing. It's the most common real failure with reasoning models, and it looks like a dozen different bugs if you don't name it.

Read your test output. 49/49 green while the app failed every single rung.

Try it

npm run install:all
npm run dev          # API :3001, UI :5173

Paste any OpenAI-compatible key in Settings, pick a concept, run the ladder. Or point Model B at your own local server and race a hosted model against it.

The repository includes the full verification suite, the mock provider, and the raw JSONL evidence behind every number in this post.

Code & more: https://www.dailybuild.xyz/project/259-one-sentence-torture-test

Building The Confidence Curve

Harish Kotra (he/him) — Sat, 19 Sep 2026 10:01:49 +0000

There is a question that every team shipping on top of reasoning models has to answer, and almost nobody answers it with a measurement:

At what point does the extra thinking stop being worth it?

A newer model ships. It reasons harder. It costs more per call and it is slower. The vendor's
benchmark table shows it winning on average. But you do not serve averages — you serve a
distribution of requests, and a single "it's better" number tells you nothing about the easy
ones. If the newer model burns 2,900 reasoning tokens to answer a question the older one nails in
175, the average is hiding a bill.

So I built an instrument to answer it: The Confidence Curve. A fixed bank of 50 questions,
two models, every answer graded in code, rendered as a 3D surface where you can see the
crossover and see the waste.

Here is what it found, and here is how it is built.

The result

diff   A acc   B acc    gap    A tok    B tok
   1    100%     60%    -40       52      175
   2    100%     60%    -40       64      230
   3     80%     40%    -40       76      285
   4    100%     60%    -40       88      340
   5     80%     80%      0      100      395
   6     40%     60%    +20      112      540   <- crossover
   7     60%    100%    +40      124      865
   8     40%    100%    +60      136     1370   <- hot but flat
   9     20%     80%    +60      148     2055   <- hot but flat
  10     40%     80%    +40      160     2920   <- hot but flat

Three things fall out of that table.

Below difficulty 6, the newer model is 40 points behind. Not level. Behind — while spending
three to four times the reasoning tokens. If your traffic skews easy, the newer model is a
straight downgrade, and no aggregate benchmark will tell you that.

The crossover is at difficulty 6, and from there up the advantage holds.

The top of the curve is where the money burns. Model B's reasoning tokens grow about 17× from
difficulty 1 to 10 (175 → 2,920) while its accuracy goes from 60% to 80%. From difficulty 7 to
10, tokens grow 3.4× (865 → 2,920) and accuracy falls from 100% to 80%.

Then the experiment that makes the point sharpest. Run the whole thing again with reasoning
switched off:

	reasoning on	reasoning off	cost
Model A	66.0%	62.0%	−4 pts
Model B	72.0%	54.0%	−18 pts

Switching deliberation off drops the newer model below the older one — 54% against 62%. Its
entire 6-point advantage was bought with thinking tokens.

There is a detail in that second run worth being precise about, because it complicates the story
rather than tidying it. Per difficulty, the crossover with reasoning off still lands at 6:

diff   A acc   B acc    gap
   1    100%     40%    -60
   4    100%     40%    -60
   5     80%     60%    -20
   6     40%     60%    +20   <- crossover, still 6
  10     40%     80%    +40

So B still overtakes A at the hard end without reasoning — it is better there, not merely
better-resourced. What reasoning buys is the easy and middle range: across difficulties 1–5, B's
average deficit widens from −32 points with reasoning to −44 without it, and that is where the
18-point total collapse comes from. The honest summary is not "B's advantage is fake". It is that
deliberation is load-bearing across most of the difficulty range, and the aggregate number hides
which part of the curve it is holding up.

The design constraints that shaped everything

Four decisions early on determined the whole architecture.

1. Grading happens in code, or it does not happen

The temptation with an LLM comparison is to have a model judge the answers. It is the wrong
tool. An LLM judge introduces a second model's biases into a measurement about models, is not
reproducible, and costs money on every run.

So the bank ships with stored answers, and grading is a normalised string comparison:

export function extractAnswer(raw: string): string | null {
  // Prefer the last ANSWER: line — the system prompt asks for it to be final,
  // and a model that restates the format earlier should be judged on its last word.
  const matches = [...raw.matchAll(/^\s*ANSWER\s*:\s*(.+?)\s*$/gim)];
  const last = matches.at(-1);
  if (last?.[1]) return last[1].trim();
  return null;
}

The system prompt forces a machine-readable last line:

You are a precise assistant. Answer the question directly. End your answer with a final line of
exactly: ANSWER: <your answer>

Then normalise() strips everything that is presentation rather than knowledge — case,
punctuation, thousands separators, currency symbols, a leading "the", a trailing full stop — and
grade() compares against the stored answer or any declared alias, with numeric equivalence so
5.0 matches 5.

The verifier unit-tests the grader against 15 hand-written cases, and five of them must be
rejected: a wrong number (16 vs 15), an off-by-one (14 vs 15), a wrong multiple-choice
letter, an empty answer, and a response with no ANSWER: line at all. A grader that only ever
returned true would pass a naive test suite and destroy the entire measurement.

2. `reasoning_content` never leaves the provider boundary

The brief said never log or display it. The reliable way to honour that is not discipline — it is
to make the leak structurally impossible. The transport parses the provider response into a
brand-new object containing only the fields the app needs:

// reasoning_content is deliberately dropped here and never referenced again.
const content = typeof message?.['content'] === 'string' ? (message['content'] as string) : '';

return {
  content,
  latencyMs,
  promptTokens: readNumber(usage, 'prompt_tokens'),
  completionTokens: readNumber(usage, 'completion_tokens'),
  reasoningTokens: readReasoningTokens(usage),
  retried: false,
};

The raw response object goes out of scope and is never returned. Only the token count survives —
read from usage.completion_tokens_details.reasoning_tokens, with a top-level fallback because
not every OpenAI-compatible provider nests it the same way.

The verifier then asserts against the committed JSONL that no reasoning text appears anywhere,
and the browser suite asserts it never reaches the DOM.

3. The crossover must be computed, never declared

The whole value of the app is that it finds the crossover. A hardcoded constant would make the
app a very elaborate way of drawing a line I already decided on.

export function findCrossover(agg: Aggregate): Crossover {
  const gaps = DIFFICULTIES.map((difficulty) => {
    const a = agg.cells.find((c) => c.difficulty === difficulty && c.slot === 'A');
    const b = agg.cells.find((c) => c.difficulty === difficulty && c.slot === 'B');
    return {
      difficulty,
      gap: (b?.accuracy ?? 0) - (a?.accuracy ?? 0),
      aAccuracy: a?.accuracy ?? 0,
      bAccuracy: b?.accuracy ?? 0,
    };
  });

  let crossover: number | null = null;
  for (let i = 0; i < gaps.length; i++) {
    const here = gaps[i];
    if (!here || here.gap <= 0) continue;
    // The advantage must hold at every higher difficulty, not just here.
    const staysPositive = gaps.slice(i).every((g) => g.gap > 0);
    if (staysPositive) {
      crossover = here.difficulty;
      break;
    }
  }

  return { difficulty: crossover, gaps, none: crossover === null };
}

The staysPositive clause is doing real work. With five samples per difficulty, one cell is
worth 20 percentage points. A single positive gap at difficulty 7 followed by a negative gap at
difficulty 8 is noise, not a crossover. Requiring the advantage to hold all the way up is what
makes the answer stable.

Note the shape of the return value: difficulty: number | null plus none: boolean. The UI has
to handle "no crossover found" as a first-class outcome, because a real run can produce one.

4. Every call is written to disk before it is shown

The JSONL audit trail is appended per call, not written at the end:

await appendRecord(runId, record);
onProgress(record);

That ordering matters for two reasons. A sweep that dies at call 60 still leaves 60 graded
records on disk, so the partial result is recoverable rather than lost. And the committed
data/runs/*.jsonl files are the evidence behind every number in the README — one line per
graded call carrying the raw answer, the stored answer and the verdict, so anyone can check the
claim instead of trusting it.

Architecture

flowchart TB
    subgraph Browser["Browser — Vite + React + TypeScript"]
        UI[App.tsx<br/>idle / running / complete / error]
        SP[Settings<br/>localStorage]
        SV[SurfaceView<br/>tooltip + overlay]
        SC[surface.ts<br/>three.js scene]
        CA[aggregate.ts<br/>live partial aggregation]
        UI --> SP
        UI --> SV
        SV --> SC
        UI --> CA
    end

    subgraph Node["Node — Hono on :3001"]
        API[app.ts<br/>routes + SSE]
        SW[sweep.ts<br/>orchestration]
        PV[provider.ts<br/>fetch, drops reasoning_content]
        GR[grade.ts<br/>normalised comparison]
        AG[aggregate.ts<br/>crossover + wasted zone]
        ST[store.ts<br/>JSONL append]
        API --> SW
        SW --> PV
        SW --> GR
        SW --> ST
        SW --> AG
    end

    subgraph Disk["Disk"]
        JSONL[(data/runs/runId.jsonl)]
    end

    subgraph External["External"]
        LLM[OpenAI-compatible<br/>/chat/completions]
    end

    UI -->|POST /api/curve| API
    API -.->|SSE: start, progress, done, error| UI
    PV -->|plain fetch| LLM
    ST --> JSONL

A 100-call sweep against a real reasoning model takes minutes. That single fact dictates the
transport: the run cannot be a request that returns a result, because nothing would be on screen
for minutes.

sequenceDiagram
    participant B as Browser
    participant A as API (Hono)
    participant S as sweep.ts
    participant P as Provider
    participant G as grade.ts
    participant D as JSONL

    B->>A: POST /api/curve {questionIds, apiKey, config, condition}
    A-->>B: 200 text/event-stream
    A->>S: runSweep(...)
    S-->>B: event: start {runId, total: 100}
    loop 50 questions x 2 models
        S->>P: POST /chat/completions
        P-->>S: content + usage.completion_tokens_details
        S->>G: extractAnswer(content), grade(...)
        S->>D: append record
        S-->>B: event: progress {done, total, record}
    end
    S->>S: aggregate, findCrossover, findWastedZone
    S->>D: append summary
    S-->>B: event: done {result}

Server-sent events, four frame types, one per graded call. Hono's streamSSE makes the server
side trivial; the client side needs a little care, because EventSource cannot issue a POST and
the sweep parameters (including the API key) belong in a body, not a query string. So the client
reads the response body directly and parses frames by hand:

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  // SSE frames are separated by a blank line.
  let boundary = buffer.indexOf('\n\n');
  while (boundary !== -1) {
    const frame = buffer.slice(0, boundary);
    buffer = buffer.slice(boundary + 2);
    boundary = buffer.indexOf('\n\n');
    const data = frame.split('\n')
      .filter((line) => line.startsWith('data:'))
      .map((line) => line.slice(5).trimStart())
      .join('\n');
    if (data) handlers.onEvent(JSON.parse(data) as SweepEvent);
  }
}

The buffer-and-boundary loop is the part that bites people. Chunks do not respect frame
boundaries — a single read() can deliver half a JSON payload, and the next one delivers the
rest. Accumulating into a buffer and only parsing complete frames is what makes it correct.

Rendering: making a 20-bar table legible

The data is two models × ten difficulties × two numbers (accuracy and reasoning tokens). A table
communicates that; it just does not communicate shape. The crossover is a place in space, and
the wasted zone is a region, and both read better as terrain.

three.js, two ridges of columns — Model A at the back, Model B at the front:

const ROW_A_Z = -2.4;
const ROW_B_Z = 2.4;
const COLUMN_WIDTH = 1.5;
const ACCURACY_HEIGHT = 8;   // world units for 100%

const height = Math.max(0.06, (cell.accuracy / 100) * ACCURACY_HEIGHT);
const t = normaliseTokens(cell.meanReasoningTokens, min, max);
const colour = rampColor(t);

The colour ramp is the load-bearing encoding. It is defined once and consumed by both the
three.js scene and the HTML legend:

export const RAMP_STOPS = [
  { at: 0.0,  color: '#1d3f5c' },  // deep cool blue — barely thinking
  { at: 0.25, color: '#2f7fa8' },
  { at: 0.5,  color: '#57a86f' },  // neutral green — the middle of the range
  { at: 0.72, color: '#e0a92c' },
  { at: 0.88, color: '#f2701d' },
  { at: 1.0,  color: '#ff3d1f' },  // hot — burning tokens
];

If the scene and the legend each carried their own copy of the ramp, they would drift the first
time someone tuned a colour, and the app would quietly start lying about its own scale. One
definition, two consumers.

Two other details worth stealing:

Normalise the colour range to the measured data, not to an absolute scale. A run where Model
A peaks at 160 reasoning tokens and Model B at 2,920 would render every A column as the same
cold blue under a fixed 0–3,000 scale. Normalising to reasoningRange(aggregate) makes the
relative burn visible, which is the actual question.

Export PNG by temporarily resizing the renderer, not by upscaling a screenshot:

exportPng(scale: number) {
  const w = container.clientWidth || 1;
  const h = container.clientHeight || 1;
  const previousRatio = renderer.getPixelRatio();
  renderer.setPixelRatio(scale);
  renderer.setSize(w, h, false);
  camera.aspect = w / h;
  camera.updateProjectionMatrix();
  renderer.render(scene, camera);
  const url = renderer.domElement.toDataURL('image/png');
  // Restore the interactive size.
  renderer.setPixelRatio(previousRatio);
  renderer.setSize(w, h, false);
  renderer.render(scene, camera);
  return url;
}

This gives a true 2x render (2520×1740 from a 1260×870 viewport) with crisp label text, rather
than a blurry upscale. It requires preserveDrawingBuffer: true on the renderer, which costs a
little performance and is worth it.

The bug that taught me the most

The wasted-compute detector was originally a comparison between the two models: flag any
difficulty where Model B spent many more tokens than Model A without being much further ahead.

It flagged difficulty 6 — the crossover itself — as wasted compute. But at difficulty 6, Model B
is 20 points ahead. Being ahead is not waste.

The bug was a definition error, not a coding error. I had written down "wasted" as a relative
concept when the actual question is about marginal return: is this model, at this difficulty,
spending meaningfully more than it did one level down for no accuracy gain? That is a question
about one model's own curve.

export const WASTED_TOKEN_GROWTH = 0.25;   // >= 25% more reasoning tokens than the level below
export const WASTED_ACCURACY_GAIN = 5;     // ...for <= 5 points of accuracy

const tokenGrowth = (here.meanReasoningTokens - prev.meanReasoningTokens) / prev.meanReasoningTokens;
const accuracyGain = here.accuracy - prev.accuracy;

if (tokenGrowth >= WASTED_TOKEN_GROWTH && accuracyGain <= WASTED_ACCURACY_GAIN) {
  out.push(difficulty);
}

On the reference run this correctly flags difficulties 2, 6, 8, 9 and 10:

difficulty 2: token growth +31.4%, accuracy gain  +0.0 pts
difficulty 6: token growth +36.7%, accuracy gain -20.0 pts
difficulty 8: token growth +58.4%, accuracy gain  +0.0 pts
difficulty 9: token growth +50.0%, accuracy gain -20.0 pts
difficulty 10: token growth +42.1%, accuracy gain +0.0 pts

Difficulty 2 is flagged, and it is worth sitting with that. Both models score 100% and 60% there
— exactly as they did at difficulty 1 — but B's reasoning tokens rose 31% to reach the same
place. That is the shape of waste at the easy end: not a catastrophe, just a model thinking
harder to arrive at the same answer.

And note that difficulty 6 is flagged on a negative accuracy gain. The crossover and the start
of the waste zone coincide here, which is a genuinely interesting reading: the level where B
first pulls ahead is also the level where its extra thinking stops converting into accuracy. The
two rules measure different things and are allowed to disagree; on this run they happen to point
at the same difficulty.

Building the surface live

The first version had a genuine flaw: a blank canvas for the entire duration of a sweep. The
surface only mounted once a run completed, so with a real provider you would stare at an empty
viewport for minutes behind a thin progress bar.

The fix was not a spinner. It was to build the surface from the records as they stream in, which
meant the client needed its own aggregator — the server only sends the aggregate at the end.

const liveAggregate = useMemo(
  () => (liveRecords.length > 0 ? aggregate(liveRecords, modelNames) : null),
  [liveRecords, modelNames],
);

The surface then distinguishes three states per column:

Unmeasured → faint outline plus a floor plinth, so the empty shape of the run is visible from the first frame
Partially measured → translucent with a bright outline, because a cell holds five samples and a column built from two of them must not look like a settled number
Finished → solid, heat-coloured

One subtlety this exposed: mid-sweep the crossover tag read "None found". Unmeasured cells
have a zero gap, so the rule correctly found no crossover and the UI confidently reported that
Model B never wins — while the sweep was still running. It now reads "Measuring…" and
explains that the crossover needs every higher difficulty measured before it can be named.

That is the same class of bug as the wasted-compute one: the arithmetic was right and the claim
was wrong. A confident wrong answer is worse than an honest "I don't know yet", and a
measurement tool has to be held to that standard harder than a normal app.

Verified with a browser harness that samples the DOM 26 times during a sweep: the canvas is
present in every sample, the call count climbs monotonically, and pixel analysis of the
screenshots confirms the surface genuinely fills in (8.8% lit pixels early → 22.7% mid-sweep).
A full 100-call sweep with live rendering completes in under a second against a fast provider,
with no dropped state.

The port collision that nearly produced a false result

This one is worth writing down because the failure mode was silent and the data looked fine.

The dev setup is the standard shape: Vite on 5173 proxying /api to the Hono server on 3001.
During development I restarted the API, and another project on the machine took over 3001. Vite
kept proxying faithfully. The web app called /api/questions, got a 404, and rendered an empty
bank. It called /api/curve, and got whatever the other project returned for that path.

Nothing crashed. The app just quietly talked to the wrong server.

Two fixes, both cheap:

The proxy target is configurable, and the server refuses to start on a port held by a
different server. It probes the port first and identifies what is there by shape:

// Probe the port. 'ours' means a healthy instance of this app; 'foreign' means
// something else is listening and the proxy would send /api to it.
const verdict = await probe(PORT);
if (verdict === 'foreign') {
  console.error(
    `Port ${PORT} is already in use by a different server.\n\n` +
    `The web app proxies /api to 127.0.0.1:${PORT}, so it would talk to that\n` +
    `other server instead of this one — and quietly show the wrong data.\n\n` +
    `  PORT=${PORT + 200} pnpm -F server dev\n` +
    `  PORT=${PORT + 100} API_PORT=${PORT + 200} pnpm -F web dev`,
  );
  process.exit(1);
}

The general lesson: a dev proxy turns a port conflict into a data-integrity bug. If your app
fetches from a localhost port, a port collision does not produce an error — it produces plausible
wrong data.

What is verified, and what is not

I want to be precise about this, because it is the part of the project most likely to be
overstated.

Verified offline, automatically — 26 checks. The full 50-question sweep runs twice (reasoning
on, then off) through the real sweep, grader, JSONL store, aggregation and crossover code,
against a deterministic offline transport whose competence curve is known in advance. The
crossover it produces is exactly the declared ground truth, the grader's 15 cases pass, the
reasoning-disabled run reports zero reasoning tokens across all 100 calls, and the JSONL contains
no reasoning text and no API key.

Verified in a browser, automatically — 12 + 11 checks. Settings persist across a reload, SSE
progress streams, the surface fills in live, tooltips report both models' numbers, PNG export
produces a real 2520×1740 download, reasoning_content never appears in the DOM, and there are
no page errors.

Not verified: real model behaviour. The offline transport is not a model. It proves the
plumbing, the grading, the crossover arithmetic and the visual encoding. Every number in the
table at the top of this post comes from that transport, whose crossover is scripted rather
than observed. Running it against real models needs an API key, and I have deliberately not
claimed results I did not measure.

That distinction is the honest core of the project. The instrument is verified. The reading it
gives on real models is the user's to take.

What I would add next

Error bars, before anything else. Five samples per cell means every cell moves in 20-point
steps. A crossover computed from 5-sample accuracies is a point estimate with no interval around
it, and presenting it without one is the weakest thing about the current design. A Wilson
interval per cell, rendered as a whisker on each column, would fix the visualisation; bootstrapping
the crossover across resampled records would tell you how often it lands on the same difficulty.

Repeat runs and variance. Run the same sweep N times and report the spread. Temperature 0 does not make a provider deterministic, and the variance between runs is probably comparable to the effects being measured — which is itself the most important thing a user could learn from
this tool.

Concurrency. Calls are strictly sequential, which is correct but slow. A bounded pool of 4 would cut a real sweep from minutes to seconds. The care needed is that latency measurements get noisier under concurrency and some providers rate-limit.

Cost in currency. Multiply reasoning tokens by the provider's per-token price and render a second surface in dollars. "Wasted compute" becomes a number a person can put in a budget.

The code is in this repository — fork it, point it at your own two models, and find out where your crossover is. It will not be at 6. It will be wherever your workload puts it, which is exactly why it is worth measuring.

Code & more: https://www.dailybuild.xyz/project/258-the-confidence-curve

Two Models, One Board: Building "Model vs Model"

Harish Kotra (he/him) — Fri, 18 Sep 2026 18:23:34 +0000

I wanted a way to show that a newer model is better than an older one that wasn't a table of numbers.

Benchmark tables are boring and, worse, they're unconvincing. Nobody feels anything reading MMLU: 78.3 → 81.1. But put two models on opposite sides of a tic-tac-toe board, enforce the rules with a real engine, and let them play a best-of-five, and you get something else entirely: an outcome you genuinely don't know in advance, a scoreboard you can argue with, and a format where the audience picks a side in the comments. The progression from an older model to a newer one stops being a delta and becomes a record.

So I built Model vs Model. Two OpenAI-compatible endpoints, three games — tic-tac-toe, Connect 4, and a five-round structured debate — one three.js board that renders every move live, and a persistent scoreboard across the whole session.

The interesting engineering isn't the games. It's that models are unreliable narrators of their own actions, and the whole thing only works if you refuse to trust them.

Architecture

The shape is deliberately boring: a browser, a dev proxy, an Express server, and a provider. All the interesting decisions live inside the server box.

┌──────────────────────────────────────────────────────────────────────────┐
│  BROWSER   Vite + React + TS  :5173                                      │
│                                                                          │
│    state machine:  idle ──► playing ──► finished / error                 │
│    three.js: one persistent scene, swaps board contents per game         │
└───────────────┬───────────────────────────────────────────▲──────────────┘
                │  POST /api/game                           │  SSE frames
                │  { game, rounds, settings }               │   event: start
                │                                           │   event: move
                ▼                                           │   event: done
┌───────────────────────────────────────────────────────────┴──────────────┐
│  VITE DEV PROXY   :5173 ──► :3001     /api/*   (unbuffered for SSE)      │
└───────────────┬───────────────────────────────────────────▲──────────────┘
                │                                           │
                ▼                                           │
┌───────────────────────────────────────────────────────────┴──────────────┐
│  EXPRESS SERVER   :3001                                                  │
│                                                                          │
│   ┌──────────────────────────────────────────────────────────────┐       │
│   │  MATCH RUNNER   match.ts                                     │       │
│   │                                                              │       │
│   │    for each game:                                            │       │
│   │      for each turn:                                          │       │
│   │        playTurn() ──► call model ──► parse ──► apply()       │       │
│   │                          ▲                    │              │       │
│   │                          │  rejected?         │              │       │
│   │                          └─── ONE retry ◄─────┘              │       │
│   │                               2nd failure = forfeit,         │       │
│   │                               turn passes, board untouched   │       │
│   └──────┬───────────────────────┬──────────────────────┬────────┘       │
│          │                       │                      │                │
│          ▼                       ▼                      ▼                │
│   ┌──────────────┐    ┌────────────────────┐   ┌───────────────────┐     │
│   │ RULES ENGINE │    │  LLM CLIENT        │   │  JSONL LOG        │     │
│   │ games/*.ts   │    │  llm.ts            │   │  log.ts           │     │
│   │              │    │                    │   │                   │     │
│   │ render()     │    │  plain fetch to    │   │  append-only      │     │
│   │ parse()      │    │  /chat/completions │   │  one record/move  │     │
│   │ apply()      │    │  no SDK            │   │  logs/matches     │     │
│   │ terminal()   │    │                    │   │  .jsonl           │     │
│   └──────────────┘    └─────────┬──────────┘   └───────────────────┘     │
└─────────────────────────────────┼────────────────────────────────────────┘
                                  ▼
                       ┌────────────────────────┐
                       │  MODEL PROVIDER        │
                       │  OpenAI-compatible     │
                       │  POST /chat/completions│
                       └────────────────────────┘

The core decision: the server owns the board

A model never sees game state. It sees a string — a rendered text board — plus a move history, and it returns a move. That move is validated before it touches anything.

That constraint is what makes the whole project tractable, and it's expressed as one generic interface:

export interface GameEngine<S> {
  id: GameId;
  name: string;
  /** Roles in play order, e.g. ['X','O']. */
  roles: [string, string];
  /** Max moves before the game is declared drawn (safety valve). */
  maxMoves: number;
  createState(seed: number): S;
  /** Plain-text board for the prompt. */
  render(state: S): string;
  /** Compact board for the wire (client renders 3D from this). */
  serialize(state: S): unknown;
  systemPrompt(state: S, role: string): string;
  userPrompt(state: S, role: string, ctx: TurnContext): string;
  /** Strict-ish extraction of the move from raw model output. */
  parse(raw: string): MoveParse;
  /** Rules validation. Never mutates; returns a new state on success. */
  apply(state: S, role: string, move: string): ApplyResult<S>;
  terminal(state: S): TerminalResult;
  /** Line shown in the move log, e.g. "X plays cell 4". */
  describe(role: string, move: string): string;
}

The generic parameter S is the whole trick. runMatch() in match.ts never knows whether it's playing tic-tac-toe or running a debate. It knows how to call a model, how to retry, how to forfeit, how to log, and how to stream — and it delegates every game-specific question to the engine. Tic-tac-toe's state is (string|null)[]; Connect 4's is a 6×7 grid with a gravity rule; the debate's is a list of {role, round, text} turns. The runner doesn't care.

Two details make this hold up. First, apply() never mutates: it returns a brand-new state on success and an error on failure. So a rejected move cannot leave a half-written board behind, because there's no board to write to. Second, terminal() is a pure query, so the runner can ask "is this over?" as often as it likes without side effects.

Adding a fourth game means writing one file. Nothing in the runner changes.

The retry/forfeit state machine

This is the part I'm happiest with, and it's the part that took the most thought.

Models produce malformed output constantly. They wrap JSON in prose, they return {"move": "four"}, they play in an occupied cell, they ignore the gravity rule. A naive implementation drops the bad move, or crashes, or — worst — applies something half-parsed and corrupts the board.

The rule I settled on: one correction retry, then a forfeit. Never silently drop a move, never corrupt state.

The retry is not just "try again". It's a constrained retry that hands the model its own rejected output, the exact reason for rejection, and the list of legal moves:

for (let attempt = 1; attempt <= 2; attempt++) {
  const messages: ChatMessage[] = [{ role: 'system', content: system }];
  if (attempt === 1) {
    messages.push({ role: 'user', content: user });
  } else {
    // The correction retry explains exactly what was wrong and what is allowed.
    messages.push({ role: 'user', content: user });
    messages.push({ role: 'assistant', content: attempts[0].raw || '(no output)' });
    messages.push({
      role: 'user',
      content: [
        `Your previous answer was REJECTED by the rules engine: ${lastError}`,
        `Legal moves right now: ${legalMoves.length ? legalMoves.join(', ') : 'none'}.`,
        'Reply again with ONLY the corrected JSON object. No prose, no explanation.',
      ].join('\n'),
    });
  }
  // ... call the model, parse, then apply()
}

Feeding the rejected output back as an assistant turn matters. It puts the model in a conversation with its own mistake rather than asking it to try again blind, and it reliably fixes the "I wrapped the JSON in prose" class of failure.

If the second attempt also fails, the turn is a forfeit. And here's the semantic that took me a rewrite to get right: a forfeit passes the turn. It does not end the game. The board is untouched, the opponent plays next, and play continues. If a game reaches the move cap without a terminal state, the side that forfeited more turns loses — but a single forfeit is a penalty, not a loss.

Every attempt is preserved. The JSONL record for a turn keeps both attempts, so a rejected move is visible forever:

{
  "moveNumber": 9, "player": "A", "model": "…", "move": "4",
  "legal": true, "illegal": false, "forfeit": false,
  "correctionAttempted": true, "correctionSucceeded": true,
  "attempts": [
    { "attempt": 1, "raw": "{\"move\":3}", "legal": false,
      "error": "column 3 is full — gravity means a disc cannot be placed there. Legal columns: 0, 1, 2, 4, 5, 6" },
    { "attempt": 2, "raw": "{\"move\":\"4\"}", "legal": true }
  ],
  "result": "A"
}

One record per move, with the full retry trail inside it.

Gravity as a testable rule

Connect 4's rule is the best demonstration because it's a rule a model can see and still violate. The board renders as text, with a legend and a full-column marker:

    0   1   2   3   4   5   6
  | . | . | . | . | . | . | . |
  | . | . | . | . | . | . | . |
  | . | . | . | . | . | . | . |
  | . | . | . | . | . | . | . |
  | . | . | . | . | . | . | . |
  | R | Y | R | Y | R | Y | R |
  full?  F

And apply() enforces it:

const row = landingRow(state, col);
if (row === -1) {
  const open = Array.from({ length: COLS }, (_, c) => c).filter((c) => !columnFull(state, c));
  return {
    ok: false,
    error: `column ${col} is full — gravity means a disc cannot be placed there. Legal columns: ${
      open.length ? open.join(', ') : 'none'
    }`,
  };
}
const rows = state.rows.map((r) => r.slice());
rows[row][col] = role;
return { ok: true, state: { rows, moveNumber: state.moveNumber + 1 } };

That error string is not decorative. It's the exact text the correction retry hands back to the model, and it's what appears in the move log with a red badge. Because it names the rule and lists the legal alternatives, the retry almost always succeeds — which is the point. The retry exists to fix recoverable mistakes, not to punish.

To prove the path is real, the mock provider I use for testing deliberately breaks the rule: on a schedule, when a column is full, it plays into that column anyway. The retry then succeeds and the whole thing shows up in the log as a two-attempt record. I didn't want a retry path that only existed in theory.

The reasoning-token trap

Two things bit me here, and both are worth knowing.

max_tokens has a floor of 900, and it is not negotiable. A move is a few tokens — {"move": 4} is about five. So setting max_tokens: 20 seems obviously right. It isn't. If the model does any hidden reasoning, the reasoning consumes the budget, the model is cut off mid-thought, and message.content comes back empty. You get a perfectly successful HTTP 200 with nothing in it. The floor is enforced in one place:

export function effectiveMaxTokens(requested: number): number {
  const n = Number.isFinite(requested) ? Math.floor(requested) : 900;
  return Math.max(900, n);
}

Reasoning tokens live in a nested usage field. Not usage.reasoning_tokens on every provider, and definitely not in the message:

const reasoningTokens: number =
  usage?.completion_tokens_details?.reasoning_tokens ??
  usage?.reasoning_tokens ??
  0;

And the rule I enforced throughout: reasoning_content is counted, never stored. The provider sends it; the client reads the usage field and discards the text. It is never logged, never put in an SSE event, never rendered. Only the token count survives, because that's the number the audience actually cares about — it's the visible cost of thinking.

I test this adversarially. The mock provider returns a long reasoning_content string on every call, and the harness asserts that the string never appears in the JSONL log or in the DOM. If someone later "helpfully" adds reasoning to the UI, the test fails.

SSE over POST

I needed the server to stream each move as it's decided. EventSource is the obvious tool and it's the wrong one: it's GET-only, and a match request carries a settings object including an API key. Putting a key in a query string is not something I'm willing to do.

So the client uses fetch with a POST body and reads the response body as a stream, buffering on the SSE frame delimiter:

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

for (;;) {
  const { value, done: streamDone } = await reader.read();
  if (streamDone) break;
  buffer += decoder.decode(value, { stream: true });

  // SSE frames are separated by a blank line.
  let sep: number;
  while ((sep = buffer.indexOf('\n\n')) !== -1) {
    const frame = buffer.slice(0, sep);
    buffer = buffer.slice(sep + 2);
    dispatch(frame, handlers);
  }
}

The buffering matters: a chunk boundary can land mid-frame, so you accumulate and only dispatch on a complete \n\n-terminated frame. The frame parser also skips comment lines, which is how the 15-second heartbeat (: ping) keeps intermediaries from timing the connection out:

for (const line of frame.split('\n')) {
  if (line.startsWith(':')) continue; // heartbeat
  if (line.startsWith('event:')) event = line.slice(6).trim();
  else if (line.startsWith('data:')) dataLines.push(line.slice(5).trim());
}

One extra thing on the server: the Vite dev proxy will happily buffer an event stream unless you tell it not to, so the proxy config forces cache-control: no-cache, no-transform on text/event-stream responses.

The three.js rendering

One persistent BoardScene owns the renderer, camera, OrbitControls, lights, and a render loop. Swapping games calls setGame(), which disposes the previous board's meshes and rebuilds — the scene itself never gets torn down. The React wrapper pushes board snapshots in; it doesn't manage three.js objects.

Tic-tac-toe pieces are extruded: the X is two crossed BoxGeometry bars, the O is a torus laid flat, both with real shadow casting and a slight emissive lift so they read against the dark plate. Winning lines glow by walking the piece's materials and pushing emissive to gold.

Connect 4 gets the animation I cared most about. Discs don't fade in — they fall, under real integration, and bounce once with damping:

// Falling discs with a single damped bounce on landing.
const gravity = -22;
for (const f of this.falling) {
  if (f.bounced) continue;
  f.vy += gravity * dt;
  f.y += f.vy * dt;
  if (f.y <= f.target) {
    f.y = f.target;
    if (!f.bounced && Math.abs(f.vy) > 1.2) {
      f.vy = -f.vy * 0.28; // bounce

Each move also nudges the camera toward the cell that was just played, then eases back to the resting framing. It's a small offset applied to the base camera position, so it composes with whatever the user has done with OrbitControls:

nudgeToward(worldPoint: THREE.Vector3, strength = 0.22): void {
  const dir = worldPoint.clone().sub(this.baseTarget);
  dir.y = 0;
  if (dir.lengthSq() < 1e-6) return;
  dir.normalize().multiplyScalar(strength * 2.4);
  this.nudge.from.copy(this.camera.position);
  this.nudge.to.copy(this.basePos).add(dir).setY(this.basePos.y + strength * 1.2);
  this.nudge.t = 0;
}

Testing without an API key

I didn't have a key while building this, and honestly that constraint improved the code. I wrote a small OpenAI-compatible provider (scripts/mock-provider.mjs) that reads the board out of the prompt and plays legal moves — it parses the "empty cells you may play" list, does win/block checks, and picks sensibly. It also breaks the gravity rule on a schedule, and returns reasoning_content on every call.

That gives me two harnesses that run with zero credentials:

npm run verify — 76 checks on the rules engine, retry/forfeit paths, and the JSONL log.
npm run ui-check — 65 checks driving the real UI in headless Chrome over the DevTools Protocol.

The principle I held to: assert against independently re-derived state, not the server's own output. When the harness checks that a tic-tac-toe game ended legally, it doesn't trust the server's outcome field. It re-reads the final board, counts X's and O's, confirms |X − O| ≤ 1, scans all eight lines for a win, and only then checks that the server's claim matches. For Connect 4 it scans every column bottom-up to prove no disc is floating. For the log it re-reads the file and counts records per game.

This caught real bugs. A server that reports "legal terminal state" while its board is subtly wrong would pass a naive test and fail this one.

The bugs I actually hit

These are the ones worth your time.

req.on('close') killed the SSE stream after one event. The first end-to-end run hung forever: the browser got event: start and then silence. The match was running fine — records were being written to the JSONL log the whole time — but every subsequent write was suppressed. The cause: I guarded writes with a closed flag set by req.on('close'). In Node, the request stream emits close as soon as its body has been fully consumed. express.json() consumes the POST body immediately, so close fired before the first move, and every write after that was silently dropped. The fix is one word:

// Listen on the RESPONSE, not the request: `req` emits 'close' as soon as the POST body
// has been consumed, which would suppress every subsequent write.
res.on('close', () => {
  closed = true;
});

Forfeit semantics: it must pass the turn, not end the game. My first implementation treated a forfeit as an immediate game loss. That's a defensible reading of "forfeit" and it's wrong for this format — it makes a single malformed response decide a match, and it makes a best-of-five meaningless when a model has one bad turn. A forfeit now passes the turn; the game continues; the move cap plus a per-seat forfeit count decides the result if no terminal state is ever reached. The harness explicitly asserts the game continued after a forced forfeit rather than aborting.

moveNumber collisions. Because a forfeited turn doesn't advance the board's own move counter, the next player's move reused the same number — two JSONL records for "move 1" in the same game. The fix was to separate the two concepts: moveNumber in the log and on the wire is a monotonic turn number, incremented on every turn including forfeits, while the board's internal counter tracks applied moves. Uniqueness is now a property the harness checks.

Records appended before the result was stamped. I wrote each record to disk as the move happened, then stamped the game's result field onto the in-memory array afterward. Append-only JSONL can't be edited, so on disk every record was missing result. The fix was to buffer a game's records in memory and append them once the outcome is known — still append-only, still one record per move, now with the result present. The tradeoff: a hard crash mid-game loses that game's records. For a game bounded at 42 moves I'll take it.

gl.readPixels returned black. My browser check read pixels straight out of the WebGL context and got zeros, which looked exactly like "the board isn't rendering." It was a false alarm: the browser clears the drawing buffer after compositing, so a read-back outside a draw call always returns black even when the frame rendered perfectly. I replaced it with a screenshot analysed pixel-by-pixel (via a small zlib-based PNG decoder), which measures what the user actually sees. I nearly "fixed" a working renderer because of this.

An unrelated process on port 3001 silently swallowed requests. Another project on my machine was listening on 3001. My Vite proxy pointed there, so requests went to that server, which returned its own health payload and 404'd the game endpoint. My first instinct was that my code was broken. Now the API port is configurable (API_PORT=3210 PORT=3210 npm run dev) and both harnesses refuse to start against an occupied port rather than reporting false passes.

An unreachable endpoint produced a slow forfeit cascade. Point the Base URL at a dead host and every call fails. My retry logic dutifully retried, forfeited, passed the turn, and did the same for the next twenty turns — eventually "completing" a match where nothing happened. A transport failure is a configuration problem, not a gameplay one, so it now aborts the match after two consecutive unreachable calls and surfaces the provider's real error text in the UI.

That last one has a general lesson: retry-and-forfeit is the right policy for bad output, and the wrong policy for broken infrastructure. Conflating them hides your own misconfiguration behind a plausible-looking game.

Playing any two models: the player library

The first version hardcoded exactly two slots: one Base URL, one key, Model A and Model B. That
is the right shape for a single comparison and the wrong shape for everything after it. You
cannot compare a hosted model against a local one, and you cannot put five models from the same
provider in a rotation without re-typing the key five times.

So the config became a two-level library:

Endpoint  ── owns the base URL and the API key
   ▲
   │ endpointId
   │
ModelEntry ── owns a model name
   ▲
   │ seatA / seatB
   │
  Seat

The reason for two levels rather than one flat list of "players" is the second use case in the
title of this section. If each player carried its own key, "one endpoint with five models" would
mean pasting the same secret five times, and rotating that key would mean editing five rows. Put
the key on the endpoint and the model list becomes free.

On the wire this stays simple. The client resolves a seat down to a single flat object before
starting a match, so the server never learns that a library exists:

// web/src/lib/settings.ts
export function resolveCompetitor(config: AppConfig, modelId: string): Competitor | null {
  const entry = config.models.find((m) => m.id === modelId);
  if (!entry) return null;
  const endpoint = config.endpoints.find((e) => e.id === entry.endpointId);
  if (!endpoint) return null;
  return {
    label: entry.label || entry.model,
    baseUrl: endpoint.baseUrl,
    apiKey: endpoint.apiKey,
    model: entry.model,
  };
}

Which means the match runner did not have to change at all. It already took a seat and a model
name; now it takes a seat and a small object that happens to include the endpoint. The generic
GameEngine<S> boundary paid for itself a second time.

Two bugs this feature introduced, both worth knowing about.

First, the API key leaked into the done event. Seats used to hold a bare model name, so
echoing seats back to the client was harmless. Once a seat held the whole competitor — key
included — that echo started shipping secrets back over the wire and into the browser. The fix
is a redaction pass, and the harness now asserts the key is absent from the payload and from
the DOM:

/** The `done` event carries the seat assignment; it must not carry the keys with it. */
function redactSeats(seats: { A: Competitor; B: Competitor }) {
  return { A: redactCompetitor(seats.A), B: redactCompetitor(seats.B) };
}

Second, a misconfigured endpoint quietly forfeited an entire match. The old transport-failure
guard only fired if no call had ever succeeded:

if (transportSuccesses === 0 && transportFailures >= 2) throw new Error(...)

With one seat working and one seat broken, transportSuccesses was never zero, so the broken
seat forfeited every single turn and the match "completed" with a lopsided score. That is a
config error wearing a gameplay costume. The fix is to count consecutive failures per seat,
because a seat that cannot be reached twice in a row is not a model playing badly — it is a
wrong URL or a bad key:

if (turn.transportError) {
  transportFailures[seat]++;
  if (transportFailures[seat] >= 2) {
    const target = seats[seat];
    throw new Error(
      `Could not reach ${target.label || target.model} at ${target.baseUrl} — ` +
        `two consecutive calls failed. Check that endpoint's base URL and API key in Players.`,
    );
  }
} else {
  transportFailures[seat] = 0;
}

The general lesson, and it is the same one as before: retry-and-forfeit is the right policy for
bad output and the wrong policy for broken infrastructure. Adding a second endpoint made the
distinction load-bearing, because now one seat can be perfectly healthy while the other is
unreachable.

The library also has to keep the app always playable. Deleting an endpoint cascades to its
models, and a seat pointing at a deleted model is repaired rather than left dangling; the final
endpoint and the last two models cannot be deleted at all. The browser harness walks that whole
path — add a second endpoint, add a third model, attach it, delete the endpoint, confirm both the
cascade and the seat repair — because "the user can break their own config" is exactly the kind
of thing that only shows up in a real browser.

What I'd build next

A fourth game to prove the engine abstraction. Nim or Reversi would each be one file.
ELO across sessions, persisted outside the JSONL log so the scoreboard survives a restart and accumulates across days.
A judge model for the debate. Right now the debate is scored by a deterministic published rubric (cap compliance, engagement, concreteness, concision) so results are reproducible. A judge model would be more interesting and less honest — worth building as an explicit alternative with the bias caveats visible.
Replay from JSONL. Every record has the seed, the move, and the board. A scrubbable replay of a past match is nearly free.
Cost and token accounting per match, since I'm already logging prompt_tokens and completion_tokens.
Parallel matches on different seeds, to turn a single game into a distribution — which is what you actually need before claiming a model is better.
A head-to-head ladder where models play every game in rotation, so the first-move advantage averages out across a schedule instead of being neutralised one game at a time.

Caveat

I built and verified this without an API key. Everything you can test locally is real: the rules engines, the retry and forfeit paths, the SSE streaming, the JSONL log, the three.js board, the two harnesses — 76 rules checks and 65 browser checks, all passing against a local OpenAI-compatible mock.

What is not verified is the request shape against the specific provider it defaults to. The defaults point at https://api.particle.ai/v1 with models deepseek-v4-flash-0731 and deepseek-v4.1-flash, but I never sent a live request to that endpoint. The client uses response_format: { type: "json_object" }, a max_tokens floor of 900, and optionally chat_template_kwargs: { "enable_thinking": false } when the "Disable reasoning" toggle is on — and whether that last field is accepted, ignored, or rejected by a given provider is exactly the kind of thing you only learn by trying it. If the provider refuses the request, the real error text appears in the UI banner rather than a generic message, which is the next best thing to having tested it.

Code & more: https://www.dailybuild.xyz/project/257-head-to-head-games

Automating LLM A/B Testing: How to Build a Cross-Model Evaluation Harness Programmatically

Harish Kotra (he/him) — Fri, 18 Sep 2026 13:45:28 +0000

If you are building LLM-powered products, you already know that benchmark leaderboards are not enough. What matters is how a model behaves on your prompts, your tone, your domain data, and your budget.

This guide is for engineers and product teams who want to build a repeatable evaluation harness that can compare several open-weight models on the same dataset, score them automatically, and choose a winner using accuracy, latency, and token cost. By the end, you will have a working mental model for building a cross-model evaluation system that runs asynchronously, stores its results locally, and uses Featherless.ai as the model execution layer.

The problem with ad hoc model testing

Most teams evaluate models in the least reliable way possible:

copy a prompt into a chat UI
compare a couple of responses by eye
maybe keep a spreadsheet
repeat the process later with a different model

That approach does not scale. It breaks down when you need to compare many rows, many models, or multiple scoring criteria at once. It also makes it hard to answer the real question:

Which model performs best on this specific dataset, under this specific cost and latency profile?

That is the problem this harness solves.

What Featherless contributes to the system

Featherless is the model execution layer for this project. Instead of provisioning GPUs, managing serving stacks, or wiring together one-off model hosts, the harness sends requests directly to Featherless’s OpenAI-compatible API and lets the platform handle model availability, concurrency limits, and serverless inference.

That matters because the evaluation harness can focus on orchestration, scoring, and reporting rather than infrastructure. The app uses four Featherless surfaces:

GET /v1/models for live model discovery and filtering
OpenAI-compatible chat completions for candidate generation and judging
GET /account/concurrency for concurrency snapshots before a run
GET /account/concurrency/stream for live concurrency telemetry during a run

What we are building

The project is a full-stack LLM evaluation dashboard built with FastAPI on the backend and Next.js on the frontend.

The workflow is straightforward:

upload a CSV dataset with prompts and expected answers
select 2 to 10 Featherless models
fan out requests concurrently across the selected models
score each candidate against ground truth with an LLM judge
store logs and results in SQLite
compare models in a dashboard ranked by weighted score

How the async fan-out works

The core performance trick is concurrent model execution. A naive loop would query one model at a time, which is slow and wastes the fact that the evaluation of each row is independent.

Instead, the harness evaluates a row by dispatching one task per selected model and waiting for them together with asyncio.gather.

tasks = [
    evaluate_model(run, row, model, semaphore)
    for model in selected_models
]
results = await asyncio.gather(*tasks)

To keep the system safe against plan limits, the app wraps model calls in a semaphore. That lets users choose how many requests can run in parallel without exceeding their Featherless concurrency budget.

parallel_model_calls = max(
    1,
    min(
        len(models) or 1,
        settings.max_parallel_models,
        int(run.get("parallel_model_calls") or 1),
    ),
)

semaphore = asyncio.Semaphore(parallel_model_calls)

This is where Featherless’s cost-based concurrency model matters. A large model can consume more concurrency units than a smaller one, so “parallel” does not simply mean “unlimited.”

How candidate outputs are collected and scored

Each candidate model receives the same prompt and system instruction. The goal is consistency, not creativity.

response = await client.chat.completions.create(
    model=model,
    messages=[
        {"role": "system", "content": "Answer the user's prompt directly. Be concise and factual."},
        {"role": "user", "content": prompt},
    ],
    temperature=0.0,
    max_tokens=768,
)

For each candidate response, the harness records:

output text
latency
prompt tokens
completion tokens
total tokens
any provider error

Then the judge model compares the candidate answer to the ground truth and returns a score from 0.0 to 1.0.

messages=[
    {
        "role": "system",
        "content": (
            "You are a strict LLM evaluation judge. Compare a candidate answer against "
            "the expected ground truth for semantic correctness. Return JSON only with "
            'shape {"score": 0.0, "rationale": "short reason"}.'
        ),
    },
    {
        "role": "user",
        "content": (
            f"Prompt:\n{prompt}\n\n"
            f"Expected Ground Truth:\n{expected}\n\n"
            f"Candidate Output:\n{candidate_output}\n\n"
            "Return only the JSON score object."
        ),
    },
]

The parser is deliberately tolerant because real models sometimes return fenced JSON, not bare JSON.

{
  "score": 1.0,
  "rationale": "The answer matches the ground truth."
}

That is important in production. If your parser assumes perfect formatting, your judge becomes a brittle point of failure.

How the dataset is structured

The harness expects a CSV with three columns:

ID
Prompt Input
Expected Ground Truth

Example:

ID,Prompt Input,Expected Ground Truth
ticket-001,"Summarize this customer complaint: The user cannot reset their password after three attempts.","The user cannot reset their password because repeated attempts triggered a lockout."
ticket-002,"What is 18% of 250?","45"
ticket-003,"Rewrite this answer to sound more professional: 'I can't help with that right now.'","I’m unable to assist with that request at the moment."

This format is intentionally simple. It works for math, summarization, rewriting, extraction, and support workflows.

A real example run

Here is the kind of comparison this harness is meant to produce.

Scenario: a support team wants to compare two models on ticket summarization quality.

Dataset: 25 rows of internal support tickets
Candidates:

deepseek-ai/DeepSeek-V4-Flash
google/gemma-4-E4B-it

Judge:

a separate judge model optimized for rubric-based evaluation

Observed result:

Model	Semantic Accuracy	Avg Latency	Total Tokens	Rows	Errors	Weighted Score
`deepseek-ai/DeepSeek-V4-Flash`	0.560	3.60s	12,749	25	10	0.442
`google/gemma-4-E4B-it`	0.393	4.63s	17,987	25	9	0.151

What this tells you:

deepseek-ai/DeepSeek-V4-Flash was the better fit overall for this dataset
google/gemma-4-E4B-it used more tokens and still scored lower
the winner was selected by a custom weighting of accuracy, latency, and token use

That is the point of the harness. A leaderboard can tell you what is popular. This tells you what works for your task.

Why the judge model is separate

A strong candidate model is not automatically a strong judge. Those are different jobs.

A good candidate model should:

generate useful task outputs
follow the prompt closely
be fast enough for your workflow
fit your cost constraints

A good judge model should:

apply a rubric consistently
compare output to reference data
be stable at low temperature
minimize variance across evaluations

That is why the harness uses a dedicated judge model rather than reusing one of the candidates. If you let the same model generate and judge, you mix two different roles and make the evaluation less trustworthy.

How the backend handles failures

The backend is designed to keep a run alive even when individual requests fail.

It catches:

connection timeouts
rate limiting
HTTP status errors
schema validation failures

That matters because distributed model inference fails in different ways:

a model can be temporarily at capacity
a model can return invalid structured output
a model can be unavailable on the current plan
the provider can change response shape

The right behavior is not to crash the entire run. It is to record the failure, continue the other model calls, and expose the issue in the final report.

How concurrency visibility helps

The /account/concurrency/stream endpoint is useful because model limits are not static. A model may be available in the catalogue but temporarily unavailable under current load.

The dashboard uses the stream to show:

current concurrency limit
used concurrency cost
number of active requests
the model currently consuming capacity
how long each in-flight request has been running

That gives the user a live view of whether the run is healthy or headed toward 429s.

Why the storage layer is local

The harness stores datasets, run metadata, and events in SQLite. That keeps the system reproducible and easy to inspect without adding a separate database dependency.

This is enough for:

dataset uploads
saved run history
per-row event logs
summary calculations
debugging failed runs

For a team workflow, that is usually the right tradeoff until you need shared multi-user access.

What to extend next

The most useful next features are:

row sampling controls, so you can test on 25 or 50 rows before spending tokens on the full dataset
run export to CSV or JSON, so results can move into spreadsheets or BI tools
historical comparisons, so teams can see whether a model is improving or regressing over time

These are practical extensions because they improve decision quality without complicating the core harness.

The useful way to think about model selection is not “which model is best?” It is:

Which model performs best on this dataset, under this latency budget, with this concurrency budget, using this scoring policy?

Featherless handles the model serving side. FastAPI handles orchestration. SQLite handles auditability. The async harness handles throughput. The judge model handles scoring. Together, they turn model evaluation from an ad hoc exercise into a repeatable engineering workflow.

Checkout Featherless.ai: https://featherless.ai/?ref=harishkotra

I made two LLMs draw the same cat. The constraint is the whole point.

Harish Kotra (he/him) — Thu, 17 Sep 2026 16:14:03 +0000

Building Blindfolded Pictionary Duel: SVG path data, three.js, and four bugs that only showed up against a real provider.

There is a specific kind of dishonesty that language models are very good at, and it is easiest to see in prose. Ask a model to describe a cat and you will get a fluent paragraph. Ask it to draw one and it has to commit to coordinates. There is no hedging in a d attribute.

That is the entire idea behind this project. Two models get one byte-identical prompt:

Draw a cat as SVG path data. Use a 0 0 400 400 viewBox. Output ONLY JSON matching this schema: {"paths": [{"d": "<svg path data>", "stroke": "<hex colour>", "fill": "none|hex", "width": <number>}], "label": "<short label>"}. Use at least 3 and at most 40 paths. Coordinates must stay inside the viewBox. Do not include any text in the drawing.

Both drawings render live, side by side, on tilted three.js planes. The prompt escalates across rounds — a cat → a cat on a bicycle → a cat riding a bicycle through a city at night — and each drawing reveals itself stroke by stroke, with a glowing pen head riding the tip of the current line.

No image model. No diffusion. No retouching. A chat model is asked for vector coordinates, and whatever comes back gets parsed and drawn.

Here is what I learned building it, including the four bugs that a mock would never have caught.

Architecture

Three pnpm workspaces. shared is the single source of truth for the wire contract, the prompt, and the round ladder, so the client and server cannot drift apart.

The server is stateless and holds no credentials. The client sends its config — base URL, key, both model names — with every request. Promise.all fires both calls concurrently, so a duel takes as long as the slower model rather than the sum.

The reason there is no .env file anywhere in this project is deliberate: configuration lives in the app's Settings panel, in localStorage, and is sent per request. There is nothing to leak from the repo and nothing to configure before the first run except pasting a key.

Bug 1: models typo the key, and I was deleting their drawings

The first version of the validator looked for path.d. If it was a string, keep it. Otherwise drop the path.

Then I read the raw log and found this, from both models, independently:

{"d: ": "M140 320 L152 296 L168 284 …"}

The value is perfect. The key is "d: " — a colon and a space instead of a closing quote. My validator dropped it. Silently. A whole building disappeared from a cityscape and nothing in the UI said so.

This is the class of bug that mocks hide, because when you write the mock you write the well-formed version. So the fix recovers aggressively:

// server/src/validate.ts
const PATH_KEY_CANDIDATES = ['d', 'd:', 'd: ', 'path', 'pathData', 'data', 'svg', 'points'];

function recoverPathData(entry: Record<string, unknown>): string | null {
  if (typeof entry.d === 'string') return entry.d;

  for (const [key, value] of Object.entries(entry)) {
    if (typeof value !== 'string') continue;
    const normalised = key.trim().toLowerCase().replace(/[^a-z]/g, '');
    if (PATH_KEY_CANDIDATES.includes(normalised)) return value;
  }

  // Last resort: any string value that looks like path data.
  for (const value of Object.values(entry)) {
    if (typeof value === 'string' && /^[Mm][\s\d.,-]/.test(value.trim())) return value;
  }
  return null;
}

Replaying the fix over 82 already-logged outputs recovered real strokes in four of them. Repairs are surfaced in the UI as a repaired badge with the reason attached, because a validator that silently fixes things is only marginally better than one that silently breaks them.

Bug 2: the reveal stalled at 38% and never moved

The hook of this app is watching a drawing appear stroke by stroke. So this was fatal:

progress frozen at 38%, state.strokeIndex stuck at 1

The reveal works by advancing a distance each frame (totalLength / duration) and painting only the newly travelled part. The partial-segment branch looked like this in spirit:

paint from segment start → up to (start + remaining)

Which is correct for one frame. But if a segment is longer than one frame's travel, the next frame redraws from the segment start again, never records how far it got, and the segment never completes. Progress freezes forever on the first long line.

The fix is one field:

interface DrawState {
  strokeIndex: number;
  segIndex: number;
  segT: number;
  /**
   * Distance already travelled INTO the current segment.
   * Without this, a segment longer than one frame's travel would restart from
   * its beginning every frame and the reveal would stall forever.
   */
  segOffset: number;
}

segOffset accumulates in the partial branch, resets on segment completion, and feeds both the progress counter and the pen head position. This bug is invisible in code review — it looks fine — and invisible in a unit test unless you happen to assert on a segment longer than one frame's travel. It took reading canvas pixels in a headless browser to find it.

Bug 3: reasoning models return nothing at all, and `max_tokens` is why

This one cost an entire model its output and took the longest to understand.

max_tokens pays for the model's thinking as well as its answer. A thinking model that runs out mid-thought does not return a truncated drawing. It returns empty content. Measured against a real provider with deepseek-v4.1-flash:

`max_tokens`	result
1,600	all 1,600 spent on reasoning, 0 paths
4,000	all 4,000 spent on reasoning, 0 paths
8,192	all 8,192 spent on reasoning, 0 paths (city scene)
16,384	✅ 8,350 reasoning tokens, then 23 paths

The symptom the user sees is a seat that sits there showing nothing, then a forfeit. The default budget of 1,600 was below the threshold at which the model can succeed at all.

The original retry policy doubled the budget: 1,600 → 3,200. Still nowhere near enough. So with reasoning on, the server now treats the configured budget as a floor:

// shared/src/index.ts
export const REASONING_MIN_BUDGET = 16384;

// server/src/modelClient.ts
function startingBudget(config: DuelConfig): number {
  return config.disableReasoning
    ? config.maxTokens
    : Math.max(config.maxTokens, REASONING_MIN_BUDGET);
}

function escalateBudget(current: number): number {
  return Math.max(current * 2, REASONING_MIN_BUDGET);
}

Below that floor a reasoning model cannot produce a drawing by construction. Honouring a small number literally would only guarantee a wasted call, so the value is raised — and the value actually sent is recorded in every logged request body, so it is auditable rather than assumed. The verification suite asserts on it.

reasoning_content is typed as unknown in the response interface and never read, logged, stored, or rendered. Only the token count is reported, from usage.completion_tokens_details.reasoning_tokens. The code that would touch the field does not exist.

Bug 4: the loading state said "nothing is happening"

With reasoning on, a single call can take two minutes. During that time the seat rendered:

Blank sheet
Pick a rung or type a subject to start the duel.

Because result is null while the request is in flight, and the component fell through to its idle overlay. The app was telling the user to start a duel they had already started.

The fix is a real phase rather than a null check:

const seatPhase: SeatPhase =
  duel.status === 'drawing' ? 'loading'
  : duel.status === 'error' ? 'error'
  : shown ? 'ready'
  : 'idle';

And a working state that admits what it is doing — spinner, ticking clock, indeterminate sweep, and a line that changes once the wait gets long:

{phase === 'loading' && (
  <div className="seat__loading">
    <span className="seat__spinner" aria-hidden="true" />
    <p className="seat__loading-title" role="status">
      {reasoningOn ? 'Thinking, then drawing…' : 'Drawing…'}
    </p>
    <p className="seat__loading-elapsed" aria-hidden="true">{elapsed.toFixed(1)}s</p>
    <div className="seat__loading-track" aria-hidden="true"><span className="seat__loading-sweep" /></div>
    <p className="seat__loading-hint">{loadingHint}</p>
  </div>
)}

I also made the reveal duration scale with density, because a 40-path cityscape flashing past in the same 3.4 seconds as a 12-path cat wastes the payoff after a two-minute wait.

Rendering: from `d` attribute to tilted plane

web/src/lib/svgPath.ts is a hand-written SVG path parser supporting M m L l H h V v C c S s Q q T t A a Z z, flattening curves into polylines. It deliberately does not use the DOM's getPointAtLength, because the reveal animation needs per-point positions to drive the pen head, and the pen head needs to be at an arbitrary fractional position along the current segment.

duelRenderer.ts paints those polylines into a 1024² canvas, maps it to a CanvasTexture on a subtly tilted PlaneGeometry, and adds a blurred offset copy as a soft shadow. Each seat tilts the opposite way; OrbitControls is enabled with a straight-on default camera. The drawing is the star — the 3D is there to make it feel like a physical sheet, not to compete with it.

The reveal paints append-only: each frame draws only the newly travelled distance, so 40 paths cost about as much as 3.

There is also a plain SVG fallback view that renders the same path data without WebGL. It is useful on machines without a GPU, and it is the honest way to show that the canvas contains nothing but the model's own coordinates.

Verification: no mocks, by design

The rule I set was: if it does not hit a real model, it is a unit test.

pnpm test — 17 unit tests over the pure functions: fence stripping, brace matching, command validation, coordinate clamping, malformed-key recovery, the 40-path cap, truncated JSON.
pnpm verify — drives the real app in a real browser and reads the same localStorage config the Settings panel writes. It has no API key of its own.

The harness configures itself through the app's own UI, and it exercises the failure paths the same way — by editing the model name and the token budget in the Settings panel, then pressing a rung. That is how it caught the budget problem: it asserted that a deliberately broken model forfeits while the other seat survives, and that the provider's real error text appears verbatim.

It measures the two drawings by reading pixels back out of the WebGL canvas and comparing 32×32 ink masks:

const c = document.createElement('canvas');
c.width = 256; c.height = 256;
const ctx = c.getContext('2d');
ctx.drawImage(webglCanvas, 0, 0, 256, 256);
const d = ctx.getImageData(0, 0, 256, 256).data;
// paper is #f5f3ec; anything meaningfully darker is ink

If the two models ever produced the same drawing, that check fails. It never has — but the assertion is there precisely because "the app renders the same image twice" is the failure mode that would make the whole project a lie.

Two things the suite taught me about my own claims

A single duel is a noisy sample. My first suite asserted on one duel per rung and flaked roughly one run in three, because two models drawing "a cat" sometimes land on a similar layout. It now samples several duels and asserts on the aggregate.

Some of my README was wrong. I had claimed that cross-model divergence grows with difficulty, because one run measured 18.4% → 42.8%. A later run measured 51.1% → 32.8%. The trend was noise. I removed the claim and documented both runs.

The finding I did not expect: reasoning kills the escalation

The product promise is that the filmstrip shows drawings getting more complex as the prompt escalates. That holds — but only with reasoning off. Across every duel in the log:

mode	Round 1 paths (A / B)	Round 3 paths (A / B)
reasoning off	19.0 / 19.6	38.5 / 39.0
reasoning on	20.6 / 24.6	23.0 / 24.0

With reasoning off, path counts roughly double: models pad toward the 40-path cap. With reasoning on, they barely move, and individual pairs invert.

Thinking models plan the scene and then draw it economically — fewer, more deliberate paths. The escalation is real but it is a property of greedy, non-thinking generation, not of the model's understanding.

I changed the suite to assert escalation only in reasoning-off runs and to print a note otherwise. A test that passes by being weakened is worse than no test; a claim that is quietly false is worse than both.

What I would tell you before you build something like this

Constrain the output format, not the content. The prompt is almost entirely about shape — the viewBox, the schema, the path count, the coordinates. The model decides what a cat looks like. That division is why the output is legible without explanation.

Log the raw response, always. Every duel appends the raw assistant content, the exact prompt, and the full request body to a JSONL file. Both of the parsing bugs I found were found by reading that file, not by reasoning about the code.

Distrust your own verification. Two of the four bugs above were in my test suite, not the app: a false "identical drawings" failure caused by empty double-forfeits comparing equal, and a harness that corrupted the user's settings when it crashed mid-run. Both are fixed — the second now restores settings in a finally and on SIGINT.

Provider behaviour is not in the docs. "A reasoning model returns empty content when it exhausts its budget" is not a documented contract anywhere I could find. It is the kind of thing you only learn by measuring, and it silently cost a whole model its output.

Try it

git clone https://github.com/harishkotra/blindfolded-pictionary-duel.git
cd blindfolded-pictionary-duel
pnpm install
pnpm dev

Open http://localhost:5173, paste a base URL and an API key into Settings, and press a rung.

Code & more: https://www.dailybuild.xyz/project/256-blindfolded-pictionary-duel

I measured how deterministic LLMs actually are, prompt by prompt. The results broke my hypothesis.

Harish Kotra (he/him) — Wed, 16 Sep 2026 05:13:43 +0000

A 3D heatmap of 320 model calls, and the four bugs that almost made it lie.

Temperature 0 is supposed to mean deterministic. Same prompt in, same bytes out, every time.

It's the setting people reach for when they need reproducible output, and it's the setting most of us assume makes an LLM behave like a function.

I wanted to know: is that actually true, and if not, which prompts break it?

So I built a small experiment. Eight prompts, two models, ten repetitions each — 160 calls per run, 320 total. Then I rendered the whole thing as a 3D heatmap where colour is determinism and height is how many reasoning tokens the model burned getting there.

Here's what came out, and what it cost me to trust it.

The design

Each bar is one prompt × one model. Colour is the determinism score:

determinismScore = count(most common sha256) / reps

That's a deliberately brutal metric. I hash the raw response text and count how many of the ten repetitions produced byte-identical output. Two responses differing by a single trailing space are different responses. If a model produces the same answer with different formatting, it scores low — and that's the point, because "the same answer" and "the same bytes" are very different guarantees for anything downstream that caches, diffs, or replays output.

Height is the mean reasoning tokens. That pairing is the whole visual argument: a bar that is tall and red is a prompt that is expensive and unstable — the worst quadrant, and the one you want to find before it finds you in production.

The prompt spectrum

Eight prompts chosen to span the range from "there is exactly one right answer" to "there is no right answer":

#	Prompt	Intent
1	List the first 12 primes, comma-separated, nothing else	Exact recall
2	What is 847 × 293? Show steps, final number on its own line	Maths
3	Output a 5×5 multiplication table as a Markdown table	Spatial / formatting
4	Explain what a hash function is in one sentence	Constrained prose
5	Draw an ASCII cat, 5 lines	Structured art
6	Write a haiku about the ocean	Creative, fixed form
7	Tell me a joke about programmers	Creative, conventional form
8	Describe a city that exists between sleeping and waking	Open creative

My hypothesis, stated before running anything: prompts 1–4 would be green (deterministic), 6 and 8 would be red, and the newer model's column would be greener than the older one's.

Two of those three predictions were wrong. More on that below.

The thing that makes or breaks the experiment: nonces

Before any of the visuals, there's a subtlety that invalidates the entire experiment if you get it wrong.

If you send the identical request ten times, a provider-side response cache will happily return the identical bytes ten times. Your heatmap then measures the cache, not the model. You'd get a beautiful, entirely green grid and conclude that LLMs are perfectly deterministic - a completely false result produced by a completely reasonable methodology.

The fix is to make every request genuinely unique while keeping the semantic content identical:

// server/src/sweep.ts
export function withNonce(base: string, nonce: string): string {
  return `${base}\n\n<!--nonce:${nonce}-->`;
}

Each call appends an HTML comment containing 8 bytes of fresh randomness. The model sees a different string every time; the task is unchanged. If a cache is in play, it can't match.

And because "unique nonce" is the load-bearing assumption, it's asserted, not assumed:

// server/src/nonce.ts
issue(): string {
  for (let i = 0; i < 1000; i++) {
    const nonce = randomHex(8);
    if (!this.seen.has(nonce)) {
      this.seen.add(nonce);
      this.issuedCount++;
      return nonce;
    }
    this.duplicates.add(nonce);
  }
  throw new Error('nonce generation failed: 1000 consecutive collisions');
}

The registry is seeded from the persisted log on startup, so uniqueness holds across restarts - not just within one process. And the audit re-checks it from the raw file afterwards. If any nonce is reused, the numbers are discarded, not caveated. I verified the check actually
fires by injecting a duplicate into a copy of the log: the audit prints FAIL — determinism numbers are invalid and must be discarded and exits 1. A check that has never failed is not a check.

What the data actually showed

Two independent runs, 160 calls each. Model A = deepseek-v4-flash-0731, Model B =
deepseek-v4.1-flash, temperature 0.

Prompt	Predicted	A det	B det	A reasoning	B reasoning
Exact recall	green	0.60	0.60	53	53
Maths	green	0.60	0.70	168	195
Spatial / formatting	green	0.70	0.90	343	328
One-sentence explanation	green	0.30	0.40	79	80
ASCII drawing	amber	0.40	0.40	216	164
Haiku	red	0.10	0.10	373	321
Joke	red	1.00	0.90	59	51
Open creative	red	0.10	0.10	1390	1598

Prediction 1: the creative prompts are red and tall — TRUE

The haiku scored 0.10 on both models. The open-creative prompt scored 0.10 on both. And the open-creative cell is the tallest bar in the grid: ~1,400–1,600 mean reasoning tokens, against ~50 for the joke.

That's the expensive-and-unstable quadrant, confirmed. The model thinks hardest exactly where it has the least reason to converge.

Prediction 2: exact recall and maths are green — FALSE

This is the interesting one.

"List the first 12 prime numbers" scored 0.60. The model never got the primes wrong. Not once. What varied was the comma spacing:

2,3,5,7,11,13,17,19,23,29,31,37
2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37

A coin flip between two typographic conventions, on a question with exactly one correct answer. The content was perfectly deterministic. The bytes were not.

Maths scored 0.60–0.70 for the same reason. The answer was always correct — 248,171, every single time. But the working was formatted three different ways: (847 × 200), 847 × 200, and 847×200.

So my mental model was wrong. The green cells aren't "closed-form questions". They're questions with exactly one natural rendering. The Markdown table (0.70–0.90) and the joke (0.90–1.00) both have a single strongly conventional form, and the model reproduces it. Primes and arithmetic have a correct answer but no canonical format, and the model free-varies across the space of acceptable renderings.

That's a more useful finding than the one I set out to confirm, and it has a direct practical consequence: if you're caching or diffing LLM output, correctness is not the property you should be relying on. A model that is right every time can still produce different bytes every time.

Prediction 3: the newer model is more deterministic — FALSE

Across both runs: Model A mean 0.494, Model B mean 0.476. A 0.018 gap. Model B was ahead in only 5 of 16 cells, with 7 ties.

There is no clear older-to-newer determinism progression in this data. I'd have liked a clean
"newer model = more reliable" story, but the honest read is that the interesting axis here is
the prompt, not the model. The spread between the best and worst prompt (1.00 vs 0.10) is
fifty times larger than the spread between the two models.

Reproducibility

I ran the whole thing twice. Pearson r = 0.93 on per-prompt determinism. Haiku and
open-creative pinned at 0.10 both times. Max per-cell drift was 0.30, concentrated in the
mid-range prompts. The shape of the landscape is stable even where individual cells move —
which is what you want from a measurement you're going to make claims from.

The architecture

┌──────────────────────────────────────────────────────────────┐
│  BROWSER  Vite + React + TypeScript        :5173             │
│                                                              │
│   App.tsx ──► Settings / Prompts / Overlay / Tooltip / Table │
│      ├──► heatmap.ts   InstancedMesh + OrbitControls + PNG   │
│      └──► color.ts     Oklab perceptual ramp                 │
└───────────────────────┬──────────────────────────────────────┘
                        │ fetch /api/*      SSE /stream, /replay
                        ▼
┌──────────────────────────────────────────────────────────────┐
│  SERVER  Hono + TypeScript                 :3001             │
│                                                              │
│   index.ts    routes, SSE, key resolution                    │
│   manager.ts  sequential engine, events, cancel, replay      │
│   provider.ts fetch → /chat/completions, retry, accounting   │
│   nonce.ts    nonce registry (asserted unique)               │
│   sweep.ts    aggregation + nonce audit                      │
│   store.ts    append-only JSONL, crash-tolerant reader       │
│   audit.ts    independent re-derivation from raw JSONL       │
└───────────────────────┬──────────────────────────────────────┘
                        │ one call at a time, never parallel
                        ▼
        POST {baseUrl}/chat/completions
        user: <prompt>

              <!--nonce:9f3ac41b7e0d2c85-->
                        │
                        ▼
        server/data/sweeps.jsonl  ──►  exports/ (PNG + JSONL + manifest)

Calls are strictly sequential

Every call runs one at a time. Parallelising would be faster and would ruin the data: latency
and token accounting get distorted by contention, and the whole point is that the numbers are
trustworthy. A 160-call sweep takes 6–10 minutes, and that's the correct trade.

Storage is append-only JSONL

Every record is appended the moment it completes. Nothing is ever rewritten. A crash loses at
most the in-flight call:

// server/src/store.ts
export async function appendLines(lines: JsonlLine[]): Promise<void> {
  if (lines.length === 0) return;
  await ensureDataDir();
  const payload = lines.map((l) => JSON.stringify(l)).join('\n') + '\n';
  await appendFile(JSONL_PATH, payload, 'utf8');
}

The reader skips a torn final line rather than corrupting the file, so a hard kill mid-write is
survivable. The JSONL is the source of truth — the UI is a view over it, and a page reload
re-attaches to a sweep in progress.

Progress streams over SSE

A long sweep must never look frozen. The server emits snapshot on connect (so a late
subscriber is never behind), then progress and cell events as each call lands, plus a 5s
heartbeat.

Reasoning text is never stored

The provider returns reasoning_content alongside the answer. I record only its token
count:

function extractReasoningTokens(json: ChatResponse): number {
  const n = json.usage?.completion_tokens_details?.reasoning_tokens;
  return typeof n === 'number' && Number.isFinite(n) ? n : 0;
}

The CallRecord type has no field capable of holding the reasoning text. Not "we don't log it" —
there is nowhere to put it. That's a structural guarantee rather than a discipline, and the
audit scans the raw file to confirm it holds.

Four bugs that would have made this lie

The interesting part of this project wasn't writing it. It was the four defects that only
surfaced by actually running the thing and checking the output, rather than assuming it worked.

1. The route that silently swallowed page reloads

/api/sweep/latest was registered after /api/sweep/:id. Hono matches in registration order,
so latest was captured as an id. Every page reload silently failed to re-attach to the running
sweep. No error, no warning — it just quietly did nothing. Fix: register the literal route first.

2. Tone mapping made the legend a liar

I had ACES filmic tone mapping enabled, which is a sensible default for pretty 3D. But this is a
data visualisation: a cell's rendered colour has to equal the colour shown in the legend. Tone
mapping was desaturating everything, so the green cells weren't the green in the ramp.

I only caught it because I wrote a pixel-level assertion — decode the exported PNG, count green
pixels, fail if there are none. After switching to NoToneMapping, green pixels went from
237 to 6,551. The bug was invisible to the eye and obvious to the assertion.

3. The CLI reported a number that was always 1.00

The headless runner printed each cell's determinism as soon as the first repetition landed. With
one sample, the most common hash is trivially the only hash — every cell reads 1.00. It looked
like perfect determinism everywhere. Fix: wait for all reps before reporting a cell.

4. A dropped event caused an infinite loop

The terminal done event was written like this:

void stream.writeSSE({ event, data: JSON.stringify(payload) });
if (event === 'done') { unsubscribe(); resolve(); }

The write isn't awaited. The stream closes the instant the handler returns, so done never
flushed. The browser's EventSource saw the connection drop and — correctly, per spec —
auto-reconnected, which restarted the replay from the beginning. Forever.

It presented as a hung UI. The fix is to await the write before resolving:

void (async () => {
  try { await stream.writeSSE({ event, data: JSON.stringify(payload) }); }
  catch { /* client already gone */ }
  if (event === 'done') { clearInterval(heartbeat); unsubscribe(); resolve(); }
})();

Four bugs, none of which threw an error. Every one of them produced plausible-looking output
that was wrong.

Verification, so you don't have to trust me

The repo ships the raw evidence and an independent audit:

pnpm audit        # re-derives every claim from the raw JSONL
pnpm verify:ui    # drives the real UI in a browser, hovers a cell, exports the PNG
pnpm verify:png   # decodes the exported bitmap and asserts it's readable

pnpm audit re-reads the JSONL and checks four things independently:

Claim	Result
Every call carries a unique nonce	PASS — 320 calls, 320 distinct nonces, 0 reused
Determinism derives from real sha256 of responses	PASS — 320/320 hashes match recomputed
Reasoning tokens from `usage.completion_tokens_details`	PASS — 320/320 well-formed
`reasoning_content` never stored	PASS — absent from every record and the raw file

The exported PNG and the JSONL live in exports/ alongside a manifest, so the claim is
auditable rather than asserted.

What I'd tell you to take away

Determinism is a property of the prompt, not the model. The spread across prompts (1.00 to
0.10) dwarfs the spread across models (0.494 vs 0.476). If you care about reproducibility,
choose your prompts accordingly — and measure, because intuition is a poor guide here. I
predicted two of my eight prompts wrong.

Correctness and reproducibility are different guarantees. A model can be right every single
time and still emit different bytes every single time. If you're caching, diffing, or replaying
output, you need the second property, and you don't get it for free at temperature 0.

Creative prompts are where the compute goes. The open-creative prompt burned ~30× the
reasoning tokens of the joke, at the same determinism score. Both are red; only one is
expensive. The heatmap makes that visible in a way a table doesn't.

And the meta-lesson: every one of the four bugs above produced output that looked fine. The
nonce assertion, the pixel assertion, and the independent audit weren't ceremony — they were the
only reason I caught them.

Run it yourself

git clone https://github.com/harishkotra/determinism-heatmap.git
cd determinism-heatmap
pnpm install
pnpm start        # → http://localhost:5173

Point it at any OpenAI-compatible endpoint via Settings — base URL, key, and both model names
are editable. The heatmap on load is replayed from disk at zero token cost; press ▶ Run sweep to take fresh measurements, or ↻ Replay to watch a finished run fill in cell by
cell without spending anything.

Change the prompts. That's where the interesting variance is.

Code & more: https://www.dailybuild.xyz/project/255-determinism-heatmap

I built a particle simulator to find out if two AI models are secretly the same model

Harish Kotra (he/him) — Tue, 15 Sep 2026 13:29:50 +0000

How a frozen particle cloud became the most convincing argument I could make about model sameness and the three bugs that nearly made it lie.

The question nobody answers with a picture

Every few weeks a new model ships with a new name, a new version number, and a benchmark table showing it is 3% better at something. The table is usually true and almost never useful. What I actually want to know is simpler and harder:

How much of this model is new?

Not "is it better" — benchmarks cover that. I want to know whether the thing I'm paying more for is a genuinely different model or the same weights wearing a new hat. That is a question about sameness, and the standard tools for answering it are all textual: benchmark deltas, eval scores, vibes.

So I built something that answers it visually. The idea is almost embarrassingly simple:

Send the same prompt to two models. Turn each answer into a cloud of particles — one particle per sampled point. Then try to morph cloud A into cloud B.

If the two answers are byte-identical, the particles do not move at all.

That frozen frame is the whole product. A cloud that locks in place, desaturates, and gets a big IDENTICAL stamp with a counter reading 0 particles moved.

It's called The Identical Twins Test, and this post is about how it works, why the visual framing matters more than I expected, and the three bugs that nearly made it lie.

Why particles, and why "nothing moving" is the point

Most data visualisations are built to show activity. A dashboard that animates is a dashboard that looks like it's working. That instinct is exactly backwards for this problem.

The finding here is an absence. Two models producing the same bytes is a non-event — it's the thing you'd skip past in a diff. If I rendered it as, say, a green checkmark, nobody would feel anything. A checkmark is a claim; a frozen cloud is evidence.

So the entire design goal was to make the absence of motion the loudest thing on screen:

The cloud locks — not slows down, locks.
It desaturates, going grey, like something powered off.
The camera stops orbiting (more on this below — it was a bug at first).
A large IDENTICAL badge stamps over the frame.
A counter reads 0 particles moved, and that zero is measured, not hardcoded.

And the signature UI element is a per-character sha256 diff strip: one tick per character position, lit where the two answers differ. On an identical run, the strip is completely dark. A second, independent confirmation that doesn't require reading a number.

The contrast case matters too. When the answers genuinely differ, the cloud explodes outward and reforms into the new shape. Because you've just watched a cloud refuse to move, watching one tear itself apart reads as violent.

Architecture

Two processes, and the split is deliberate.

┌──────────────────────────────────────────────────────────────────────────┐
│  Browser — localhost:5173                                                │
│                                                                          │
│  ┌────────────────────────┐   ┌───────────────────────────────────────┐  │
│  │  React shell           │   │  WebGL stage                          │  │
│  │  App.tsx               │   │  particles.ts                         │  │
│  │   • phase state machine│   │   • 220,000-particle pool             │  │
│  │   • run history        │   │   • ShaderMaterial + morph            │  │
│  │   • JSON export        │   │   • lock() / morphTo()                │  │
│  └───────────┬────────────┘   └───────────────▲───────────────────────┘  │
│              │                                │ SampleResult             │
│              │                    ┌───────────┴───────────┐              │
│              │                    │  sampler.ts           │              │
│              │                    │  text → canvas → pts  │              │
│              │                    └───────────────────────┘              │
│              │  fetch('/api/compare')      ← Vite dev proxy              │
└──────────────┼───────────────────────────────────────────────────────────┘
               │
┌──────────────▼───────────────────────────────────────────────────────────┐
│  Node server — localhost:3001                                            │
│                                                                          │
│  index.ts            POST /api/compare   POST /api/test-connection       │
│  model-client.ts     callModel(), testModel()                            │
│  hash.ts             sha256(), newRunId(), newNonce()                    │
└──────────────┬───────────────────────────────────────────────────────────┘
               │  fetch POST {baseUrl}/chat/completions
               ▼
     ┌──────────────────────────────────────────────┐
     │  Any OpenAI-compatible provider              │
     │  default: https://api.particle.ai/v1         │
     └──────────────────────────────────────────────┘

Why is there a server at all? Three reasons, and I want to be explicit because "why isn't this a static site" is the first question anyone asks:

The verdict is computed server-side. hashEqual is decided from the response bytes on the server, so the client cannot accidentally — or conveniently — misreport it.
The API key is never bundled. It lives in localStorage, goes to localhost:3001, and gets forwarded from there.
CORS. Most providers don't send permissive CORS headers, so a browser-only version dies on the first request.

Stack: Vite 8 + React 19 + TypeScript 5.9 on the front, Express 5 on Node on the back, three.js for the particles. No component library, no state manager, no CSS framework, no OpenAI SDK.

That last omission is deliberate. The provider call is a plain fetch to /chat/completions, which is what makes the app work against any OpenAI-compatible endpoint — particle.ai, OpenAI, Groq, Together, a local llama.cpp server — by changing one text field. An SDK would have bought me convenience and cost me the entire "works with anything" property.

The verdict is a byte comparison

Here's the core of it:

// server/index.ts
const [a, b] = await Promise.all([
  callModel('A', { ...shared, model: modelA }),
  callModel('B', { ...shared, model: modelB }),
]);

// The verdict is a byte comparison, not a similarity score.
const hashEqual = a.sha256 === b.sha256;

Both models get the same system prompt, the same user message, and temperature 0. Temperature 0 matters: the app is asking a question about determinism, so a divergence should be a real difference between the models rather than sampling noise.

Note what this is not: there's no embedding distance, no fuzzy matching, no similarity threshold. Identical bytes or nothing. That's a much stronger claim, and it's the one worth making visually.

Bug #1: the nonce, or how a cache could fake the entire result

This is the bug that would have destroyed the project's credibility, and it's the one I'm most glad I caught before shipping.

If the provider caches responses, sending the same prompt twice can return the same bytes twice — even when the two models would have answered differently. That manufactures an IDENTICAL verdict out of nothing.

Sit with how bad that is. The app's entire purpose is to detect sameness. A cache makes it detect sameness falsely. Every "finding" it produced would be an artifact of the infrastructure, and the more convincing the visual, the more effectively it would be lying.

The fix is a fresh random nonce on every call:

// server/hash.ts
export function newNonce(): string {
  return randomBytes(4).toString('hex');
}

// server/index.ts — the nonce is part of the user message
const userPrompt = `${prompt}\n\n[nonce ${nonce}]`;

Two calls can never share a cache key. The nonce is shown in the UI and included in the JSON export, so any run can be audited. I verified this by running the same prompt five times: five distinct nonces, five identical verdicts, one distinct digest. The stability is real, and it isn't a cache.

Bug #2: the camera was moving

The first version froze the particles. It looked right. I was happy with it.

Then I measured it. The locked cloud drifted 0.77 pixels over 2.5 seconds.

The particles weren't moving — but the camera was still orbiting. So the frame was changing, which means a viewer watching the "frozen" cloud sees motion, and a viewer who sees motion concludes something happened. Which is precisely the opposite of the claim.

"Nothing moves" has to be literally true, so lock() now stops the camera too:

lock(): void {
  this.morphing = false;
  this.phase = 'identical';
  this.material.uniforms.uFreeze.value = 1;
  this.material.uniforms.uSaturation.value = 0.55;
  this.material.uniforms.uBurst.value = 0;
  this.material.uniforms.uProgress.value = 0;
  this.targetPositions.set(this.basePositions);
  // Stop the camera too. "The particles do not move" has to be literally true,
  // so nothing in the frame is allowed to drift — including the viewpoint.
  this.orbitSpeed = 0;
  this.movedCount = 0;
  this.peakDisplacement = 0;
}

Drift after the fix: 0.064 px over 2.5 seconds. Under a tenth of a pixel. No measurable motion.

There's a subtlety in the shader worth calling out. I wanted a gentle breathing motion so a frozen cloud doesn't look like a dead render — but the breathing must not change any particle's position:

// A little breathing motion so a frozen cloud still reads as alive, but the
// particles' *positions* never change when frozen.
float breathe = sin(uTime * 0.9 + aSeed * 6.2831) * 0.06 * (1.0 - uFreeze);
displaced += dir * breathe;

The (1.0 - uFreeze) term zeroes the breathing the instant the cloud locks. The cloud still shimmers very slightly via point size, but no particle moves.

Bug #3: measuring the wrong thing

Here's a trap worth knowing if you ever test WebGL.

The obvious way to verify the cloud rendered is to call gl.readPixels on the canvas and count lit pixels. It returns all zeros. The drawing buffer isn't preserved after compositing, so by the time your test code looks, there's nothing there.

My first test suite confidently reported "0 lit sample points" on a canvas that was visibly full of particles. I spent a while debugging a renderer that was working perfectly.

The fix: measure from actual screenshots. That's how I got the drift numbers above — capture frames, diff them, compute the cloud's centroid movement. It's slower and uglier than reading a buffer, and it's the only thing that measures what the user actually sees.

The particle system

A fixed pool of 220,000 particles in a single THREE.Points with a custom ShaderMaterial.

Why a fixed pool

Reallocating geometry per run means a GPU buffer upload on every comparison and GC hitches during the morph — exactly when the animation has to be smooth. A fixed pool means the only per-run work is writing into two Float32Arrays.

Unused particles aren't hidden with a draw-range trick. They're pushed 90 world units out of frame, which keeps the shader branch-free:

// Particles that do not exist in the incoming layout are pushed out of frame
// rather than left sitting on top of the new text.
float inactive = 1.0 - aActive;
base += normalize(base + vec3(0.001)) * inactive * 90.0;

The morph

Every particle interpolates from its A position to its B position, with a radial burst peaking at the midpoint:

vec3 base = mix(position, aTarget, uProgress);

// Burst impulse: radial kick, peaked at the midpoint of the morph.
vec3 dir = normalize(base + vec3(0.001));
float wobble = hash(vec3(aSeed * 91.7, aSeed * 13.3, uTime * 0.0));
float impulse = uBurst * uBurstRadius * (0.45 + 0.55 * wobble);
vec3 displaced = base + dir * impulse;

The CPU drives two envelopes — easeInOutCubic for position, sin² for the burst:

const t = Math.min((now - this.morphStart) / this.morphDuration, 1);

// easeInOutCubic: slow out of A, fast through the middle, settle into B.
const eased = t < 0.5 ? 4 * t * t * t : 1 - Math.pow(-2 * t + 2, 3) / 2;
this.material.uniforms.uProgress.value = eased;

// Burst envelope peaks at the midpoint of the morph.
const burst = Math.sin(Math.PI * t) ** 2;
this.material.uniforms.uBurst.value = burst;

The wobble term is what stops the burst looking like a perfect sphere inflating. Each particle gets a deterministic per-seed multiplier between 0.45 and 1.0, so the cloud frays instead of expanding uniformly. That one line is most of the difference between "impressive" and "cheap".

The counter is a measurement

0 particles moved is the headline number, so it can't be a hardcoded zero. It's computed by comparing every particle's destination against its origin:

for (let i = 0; i < POOL_SIZE; i += 1) {
  const dx = /* … */;
  if (dx * dx + dy * dy + dz * dz > 0.0004) moved += 1;
}
this.movedCount = moved;

The epsilon absorbs float noise from normalisation. If the layouts are genuinely identical, every particle lands exactly where it started and the count is genuinely zero. If I ever break the morph, the counter will tell on me.

Turning text into particles

sampler.ts does the boring-but-critical work:

Draw the text into an offscreen 1400×760 canvas, wrapped and centred.
Read pixels back, sample on a 4px grid; every point where red > 110 becomes one particle target.
Normalise into centred world space of height 24 units, with ±0.8 of z-jitter for depth.

The bug that measurement caught and review never would

The first version used a fixed font size. It looked fine in code. Then I measured particle counts per preset:

Answer	Particles (before)	Particles (after)
`"Paris"`	48 — an empty frame	4,044
A tall column of 20 digits	145 — a faint streak	2,206

A five-character answer produced 48 particles. The cloud was effectively invisible, and the app looked broken for exactly the short, punchy prompts a user is most likely to try first.

Fixed by fitting the text to the frame:

// Largest font that fits, tried from biggest to smallest.
const FONT_CANDIDATES = [150, 120, 96, 76, 60, 48, 38, 30, 24, 19, 15, 12];

// Then: if the first pass came back thin, resample on a finer grid.
const MIN_PARTICLES = 2_500;
const finerStep = Math.max(1, Math.floor(step * Math.sqrt(count / MIN_PARTICLES)));

Every preset now lands between roughly 2,200 and 7,400 particles.

One subtlety I nearly got wrong: the text block is centred as a whole, not line by line. Per-line centring is the obvious implementation and it destroys the internal alignment ASCII art depends on — a cat drawn in ASCII comes out with its ears in the wrong place. Since ASCII art is one of the most visually satisfying prompt types here, that mattered.

Not leaking chain-of-thought

Reasoning models can return a reasoning_content field alongside content. That's chain-of-thought, and it has no business in a log, an HTTP response, or the DOM.

The defence is structural, not a filter. The field doesn't exist in any type in the codebase, and the one function that reads a completion never looks at it:

interface ChatCompletionResponse {
  model?: string;
  choices?: Array<{
    message?: {
      content?: unknown;
      // reasoning_content is intentionally NOT read. See extractContent().
    };
    finish_reason?: string | null;
  }>;
  usage?: {
    completion_tokens_details?: { reasoning_tokens?: number };
  };
}

Only the token count crosses the wire, from usage.completion_tokens_details.reasoning_tokens. There's nothing to leak because nothing ever reads it. I verified this with a canary string in a test provider: it never appeared in any response, log, or DOM node.

A filter would have been the obvious approach and the wrong one — filters fail open when someone adds a new code path. A field that doesn't exist can't be accidentally serialised.

"Test connection" — the button that saves the first five minutes

The app requires your own API key. That means the very first thing a new user does is type a key and hit Run, and if anything is wrong — typo'd key, missing /v1, wrong model name — they get a failure and no idea which of three things caused it.

So there's a Test connection button that sends one tiny request (16 max tokens, costs almost nothing) to each model and classifies the failure:

if (status === 401 || status === 403) {
  return 'The API key was rejected. Check that the key is complete and belongs to this base URL.';
}
if (status === 404) {
  if (lower.includes('model')) {
    return `The provider does not recognise the model "${model}". Check the model name.`;
  }
  return 'The endpoint was not found. Check the base URL — it should end in /v1 for most providers.';
}

A bad key and a bad model name are different problems with different fixes. Conflating them into "request failed" wastes the user's time at the exact moment they're deciding whether your app works.

Two details I'm happy with:

The result clears the moment you edit any connection setting. A stale "OK" can never describe a config you've since changed — a small thing that prevents a genuinely confusing failure mode.
Probes run concurrently, so testing takes as long as the slower model, not both.

What I'd tell someone building something similar

Make the absence loud. The instinct to visualise activity is strong and often wrong. If your finding is a non-event, the design problem is making that non-event feel like a verdict, and that's a harder and more interesting problem than adding more motion.

Measure the thing you're claiming. "The particles don't move" is a claim, and 0.77px of camera drift made it false. Every headline number in this app — the moved count, the drift, the particle counts — is measured rather than asserted. It caught three bugs that looked fine in review.

Structural safety beats filters. Not having reasoning_content in a type is stronger than scrubbing it before logging. Filters fail open; absent fields can't be serialised by accident.

Beware the bug that fakes your own finding. The nonce exists because a cache could have manufactured IDENTICAL verdicts. When your app's purpose is detecting a specific condition, ask what could produce that condition falsely — that's where the credibility risk lives.

Try it

The app is open source, runs locally in two commands, and works against any OpenAI-compatible provider.

git clone https://github.com/harishkotra/token-morph
cd token-morph
npm install
npm run dev

Open localhost:5173, add your key in Settings, hit Test connection, then pick the "Recursion, one sentence" preset and watch a cloud refuse to move.

The most dramatic preset is "Count to 20, one per line" — the two models lay the numbers out so differently that the cloud's silhouette barely survives the morph. The most subtle is "Ocean in 200 words", where the shape holds and only the density changes.

If you find a model pair that's byte-identical on something it really shouldn't be, I'd love to see it.

Code & more: https://www.dailybuild.xyz/project/254-token-morph

I Built a Drag Race for LLMs, and the Loser Is Usually the Better Model

Harish Kotra (he/him) — Mon, 14 Sep 2026 14:35:17 +0000

A technical deep-dive into The Reasoning Tax Race — streaming SSE, GPU particle exhaust, and making an invisible cost visible

There's a chart that every team shipping LLM features has looked at. It has two bars: one for the old model, one for the new one. The new bar is taller. The caption says something like "improved reasoning."

What the chart doesn't show you is the invoice.

Newer reasoning models think before they answer. They emit hundreds of tokens you never see, you never read, and you absolutely do pay for. On a benchmark table that's an abstraction. In production it's your latency budget and your margin.

I wanted to make that cost visible as a physical object — something you could watch and feel rather than read in a table. So I built a drag race.

The premise

Two models. One prompt. Identical bytes to both, fired at the same instant. Each gets a
lane and a rocket.

Rocket position = tokens emitted
Exhaust plume = every token, as a particle
First across the finish line = winner

Then the twist: the slower model is usually the better one, and its exhaust trail is
visibly, dramatically longer. That trail is the reasoning tax.

The rule I set for myself was simple and non-negotiable: everything must be genuinely
measured live. No fake timers, no pre-baked token counts, no simulated latency. If a
model stalls, its rocket stops and its plume stops growing. If I couldn't measure it, it
didn't go on screen.

That constraint turned out to be the entire engineering problem, and it produced the
most interesting decisions in the project.

Architecture

┌──────────────────────────────────────────────────────────────────────────┐
│  BROWSER  ·  :5173                                                       │
│                                                                          │
│   React 18 + TypeScript                                                  │
│   ┌────────────────┐  ┌─────────────────┐  ┌──────────────────────────┐  │
│   │  Control deck  │  │   Lane HUDs     │  │   Reveal panel           │  │
│   │  prompt+presets│  │ tokens/reasoning│  │  answers · latency ·     │  │
│   │  Start / Reset │  │ stopwatch · TTFT│  │  reasoning tok · sha256  │  │
│   └────────────────┘  └─────────────────┘  └──────────────────────────┘  │
│           │                    ▲                        ▲                │
│           │ useRace()          │ live telemetry         │ outcome        │
│           ▼                    │                        │                │
│   ┌──────────────────────────────────────────┐          │                │
│   │  arrivalsRef  ──drainArrivals()──┐       │          │                │
│   └──────────────────────────────────┼───────┘          │                │
│                                      ▼                  │                │
│   ┌──────────────────────────────────────────────────┐  │                │
│   │  three.js RaceScene   (own rAF loop)             │  │                │
│   │   • 2 rockets      • GPU particle exhaust        │  │                │
│   │   • starfield      • finish gate + camera rig    │  │                │
│   └──────────────────────────────────────────────────┘  │                │
└──────────────────────────────┬───────────────────────────────────────────┘
                               │
              POST /api/race   │   SSE: start · first-token · tick ·
              (proxied by Vite)│        reasoning · retry · done · finish
                               ▼
┌──────────────────────────────────────────────────────────────────────────┐
│  NODE SERVER  ·  :3001   (Express + TypeScript)                          │
│                                                                          │
│   POST /api/race ──► runRace(prompt, config, emit)                       │
│                        │                                                 │
│                        ├──► runModel('A', modelA, …) ─┐  Promise.all     │
│                        └──► runModel('B', modelB, …) ─┘  (concurrent)    │
│                                    │                                     │
│                        streamOnce() parses provider SSE,                 │
│                        counts tokens, never stores reasoning text        │
└────────────────────────────────────┬─────────────────────────────────────┘
                                     │  fetch(…, { stream: true })
                                     ▼
              ┌────────────────────────────────────────────┐
              │  OpenAI-compatible provider                │
              │  POST {baseUrl}/chat/completions           │
              └────────────────────────────────────────────┘

The stack is deliberately boring: Vite + React + TypeScript + three.js on the
front, Node + Express + TypeScript on the back. No database, no auth, no state
library, no model SDK. Model calls are plain fetch against the OpenAI-compatible
/chat/completions endpoint — which means pointing the whole thing at Ollama, vLLM, or
OpenRouter takes seconds and zero code changes.

Making the visual be the data

This is the decision the project lives or dies on.

It would have been easy to draw two rockets moving at speeds proportional to
tokens/second, or to give each lane its own scale so both look dramatic. Both would
have been lies. The gap between the rockets has to mean something.

So both lanes share one unitsPerToken:

// web/src/three/RaceScene.ts

private tokenToWorldX(tokens: number): number {
  return START_X + tokens * this.unitsPerToken;
}

One shared scale means the horizontal distance between the rockets is always exactly
proportional to the difference in tokens emitted. It's not decoration. It's the data,
rendered.

But a fixed scale has a problem: you don't know in advance how many tokens a model will
emit. Too small and the rockets fly off screen; too large and they never leave the
start line. So the scale eases while racing, keeping the leader framed.

Then comes the moment that makes the whole thing work:

/** Plant the finish line at the first finisher's token count and freeze scale. */
plantFinish(tokenCount: number) {
  const distance = Math.max(40, tokenCount);
  this.targetUnitsPerToken = FINISH_X / distance;
  this.unitsPerToken = this.targetUnitsPerToken;
  this.frozen = true;
  this.finishGate.visible = true;
  this.finishGate.position.x = FINISH_X;
  this.finishGate.scale.set(1, 0.01, 1);
}

The instant the first model finishes, two things happen: the scale freezes, and the
finish line is planted at that model's token count.

The slower model is still emitting. It keeps driving forward — straight past the finish
line, trailing a plume that keeps growing. The overshoot is drawn to scale. The
overshoot is the tax.

This is why the visual argument works where a bar chart doesn't. You don't read that
model B cost more. You watch it lose the race and then keep going anyway.

The exhaust is not a particle effect

Here's where "genuinely measured" got hard.

My first implementation spawned exhaust on a timer, with some randomness for texture. It
looked great. It was also completely fake — a model that stalled mid-generation would
keep puffing smoke like nothing was wrong. That's a lie told by a visual, which is worse
than a lie told by a number, because you feel it instead of reading it.

The fix was to make the render loop consume real token arrivals. The hook buffers them:

// web/src/hooks/useRace.ts

/**
 * Real token arrivals buffered between frames. The scene drains this each
 * frame and spawns exactly that many exhaust particles — so if the model
 * stalls, the plume stops growing.
 */
const arrivalsRef = useRef<Arrival[]>([]);

const drainArrivals = useCallback(() => {
  if (arrivalsRef.current.length === 0) return [];
  const drained = arrivalsRef.current;
  arrivalsRef.current = [];
  return drained;
}, []);

…and the scene spawns only from what it drained:

// web/src/three/RaceScene.ts

// --- exhaust: spawn ONLY from real token arrivals ---
// Each arrival carries the number of tokens that landed since the last
// frame. A stalled model produces no arrivals, so its plume stops growing.
if (s.arrivals.length > 0) {
  for (const arrival of s.arrivals) {
    const rt = s.lanes[arrival.lane];
    // One particle per token, plus a little extra while the rate is high so
    // a fast model reads as a denser plume rather than a dotted line.
    const density = 1 + Math.min(2, arrival.rate / 60);
    const count = Math.max(1, Math.round(arrival.count * density));
    this.spawnExhaust(arrival.lane, Math.min(count, 220), rt.tokens, arrival.rate);
  }
  s.arrivals.length = 0;
}

Now the plume is a readout. Token arrival rate literally becomes particle spawn
rate. A fast model reads as a dense, hot plume; a stalling model goes quiet.

Plume size also scales with cumulative tokens, so the loser's trail is both longer and
fatter:

// Plume grows with cumulative tokens: more tokens -> fatter, longer-lived.
const plumeScale = 1 + Math.min(1.6, cumulativeTokens / 900);
const rateBoost  = 1 + Math.min(1.2, rate / 90);

Doing it at 60fps

Thousands of particles per lane, spawned continuously, with zero per-frame allocation.

One THREE.Points per lane with a fixed 4000-particle pool and a ring-buffer
cursor. Dead particles aren't removed — they're pushed outside the clip volume in the
vertex shader and collapsed to zero size:

float age = uTime - aBirth;
float t = aLife > 0.0 ? age / aLife : 2.0;

if (t < 0.0 || t > 1.0) {
  // Dead particle: push it outside the clip volume and collapse it.
  gl_Position = vec4(0.0, 0.0, -2.0, 1.0);
  gl_PointSize = 0.0;
  vAlpha = 0.0;
  return;
}

Recycling is a modulo on an integer cursor. The GPU does the ageing, the drift, the
expansion, and the fade — the CPU only writes new particles into the ring.

Four bugs that taught me something

1. The SSE stream that died after one event

This one cost me an embarrassing amount of time, and the fix is a single word.

The server was doing:

req.on('close', () => { closed = true; });

In Node, req emits 'close' when the request body has been fully consumed — not
when the client disconnects. Since the POST body is tiny and arrives instantly, closed
flipped to true almost immediately, and my emit() guard silently swallowed every
event after start. The race looked like it hung. Nothing errored. Nothing logged.

The fix:

// NOTE: listen on the RESPONSE, not the request. `req` emits 'close' as soon
// as the POST body has been fully consumed, which would mark the stream dead
// before a single race event was written.
res.on('close', () => {
  closed = true;
  clearInterval(heartbeat);
});

If you're building SSE on Express, this is the bug you will hit. Two more things worth
copying: a heartbeat comment every 15 seconds so proxies don't idle the connection out
during a long think, and headers that stop anything from buffering the stream:

res.writeHead(200, {
  'Content-Type': 'text/event-stream; charset=utf-8',
  'Cache-Control': 'no-cache, no-transform',
  Connection: 'keep-alive',
  'X-Accel-Buffering': 'no',
});

2. `EventSource` can't POST

The browser's EventSource API only does GET. I need to send a prompt body. So the
client uses fetch plus a ReadableStream reader and parses SSE frames by hand:

const reader = res.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  // SSE frames are separated by a blank line.
  let sep: number;
  while ((sep = buffer.search(/\r?\n\r?\n/)) !== -1) {
    const frame = buffer.slice(0, sep);
    buffer = buffer.slice(sep + (buffer[sep] === '\r' ? 4 : 2));
    // …parse `event:` / `data:` lines, dispatch
  }
}

Note the buffer-and-split: chunks arrive at arbitrary boundaries, so a frame can be
split mid-JSON. You must accumulate and only parse on the blank-line terminator. And
flush any trailing frame after the loop, because the last one may lack a terminator.

3. An empty response is a budget problem, not a refusal

A reasoning model can spend its entire max_tokens thinking and return empty
content. To a naive client that looks like a broken model. It's actually a
misconfigured budget.

So: retry once with double the budget.

// Attempt 1, then (only if content came back empty) one retry with a
// doubled budget. An empty response is a budget problem, not a refusal.
for (let attempt = 1; attempt <= 2; attempt++) {
  attempts = attempt;
  maxTokensUsed = attempt === 1 ? config.maxTokens : Math.min(config.maxTokens * 2, 4000);
  // …
}

The subtle part is drift. If the retry builds its own request, it will eventually
diverge from the first call — someone adds a field to one path and not the other. So
both attempts build from one shared factory:

// Shared callbacks so the retry path can never drift from the first call.
const attemptOptions = (): AttemptOptions => ({ /* … */ });

4. Counting reasoning without ever reading it

This one is an ethics decision as much as a technical one.

Reasoning traces are the model's private scratchpad. They can contain the user's data,
half-formed (and wrong) intermediate conclusions, and content that was never meant to be
shown. I need the cost of reasoning. I do not need, want, or store its content.

So the server tests for presence, increments a counter, and throws the text away:

// COUNT reasoning, never capture it.
if (typeof delta.reasoning_content === 'string' && delta.reasoning_content.length > 0) {
  reasoningChunks += 1;
  opts.onReasoning(reasoningChunks);
}

There are exactly two references to reasoning_content in the entire server: that
line and the doc comment above it. The authoritative number comes from usage:

const rt = usage?.completion_tokens_details?.reasoning_tokens;
if (typeof rt === 'number') reasoningTokens = rt;

If a provider omits usage, it falls back to the live chunk count — so the HUD never
shows a lie. And npm run ui-check asserts that reasoning_content never appears in
the DOM.

Proving it isn't a nice animation

The whole value of this project rests on one claim: it measures something real. So the
claim has to be falsifiable. I wrote three scripts, no test framework.

`npm run verify`

Drives the live endpoint and asserts the two presets produce visibly different
races:

━━━ DIVERGENCE ━━━
  ✓ lanes visibly separate (token gap > 20)          (gap = 38)
  ✓ slower model has the fatter plume                (loser 34 vs winner 21)
  ✓ outputs differ                                   (identical = false)
  ✓ reasoning token counts read from usage           (A=0 B=50)

━━━ TIE ━━━
  ✓ finish near-simultaneously (< 1500ms)            (gap = 0ms)
  ✓ IDENTICAL OUTPUT                                 (identical = true)

  ✓ the two prompts produced VISIBLY DIFFERENT races

If both prompts ever produce identical-looking races, the app has stopped measuring
anything and this script says so.

`npm run ui-check` — and a measurement bug worth reading

Headless Playwright. It samples the HUD mid-race to prove counters advance live, and
analyses screenshot pixels to confirm the slower lane really renders the fatter plume.

My first version counted cyan and magenta pixels in the screenshot. It failed
constantly, and the reason is a good lesson in measuring things:

The glowing rails are the same colours as the rockets. Worse, lane A's rail sits
nearer the camera, so it's always bigger. The measurement was dominated by a static
piece of scenery, not the exhaust.

The fix: locate each lane's rail row (the densest run of that colour in its half of the
screen), then count only pixels above it — because exhaust drifts upward and the
rail does not.

run	winner plume	loser plume
1	1,868 px / 69 px tall	48,163 px / 172 px tall
2	4,039 px / 69 px tall	29,673 px / 143 px tall
3	2,264 px / 21 px tall	48,282 px / 135 px tall

Three consecutive runs, consistently ~10× difference. Now the test measures the thing it
claims to measure.

`npm run mock`

A fake OpenAI-compatible provider so the whole pipeline is testable offline: a "slow
thinker" that streams reasoning_content before content, a model that returns empty
content on its first call so the retry path is observable, and a tie mode where both
models converge on identical text and finish together.

This is how I verified the retry, the reasoning counter, and the IDENTICAL OUTPUT
verdict without spending a cent.

What I'd tell you to build next

The single highest-impact addition would be a cost ticker. Reasoning tokens are
billed. Multiply by the model's real price and put "the tax cost $0.0043" next to the
plume. That converts a metaphor into a number a team can act on.

After that: run each prompt 5× and show the median and spread. One race is an
anecdote; five is a measurement. A single race can mislead — sampling variance is
real, and the app should be honest about that.

And the most interesting experiment the architecture already supports: race the same
model with reasoning on versus off. The Disable reasoning flag is already there. The
quality delta is the honest value of the tax.

Try it

git clone https://github.com/harishkotra/reasoning-tax-race.git
cd reasoning-tax-race
npm install
cp .env.example server/.env     # paste your key
npm run dev                     # → http://localhost:5173

Then run Haiku about rain, and watch the magenta rocket lose the race and keep going.

Everything is measured live. Nothing is mocked. And if you find a race that surprises you, open an issue with the exported JSON — that's the most useful bug report this project can receive.

Code & more: https://www.dailybuild.xyz/project/253-reasoning-tax-race

Why Your AI Agents Keep Calling the Wrong Tools (And How We Built Toolsmith to Fix Them)

Harish Kotra (he/him) — Sun, 13 Sep 2026 18:22:14 +0000

If you have built an autonomous AI agent using OpenAI, Claude, LangChain, or CrewAI, you have inevitably encountered this scenario:

Your agent needs to look up a customer order. Instead of calling fetch_customer_order(order_id), it calls search_database(query="order"). Or worse, it hallucinates an argument like status="completed_order_v2" because the schema simply said "status": { "type": "string" } without defining permitted enum values. Or it enters an infinite loop, toggling back and forth between two sibling tools that do almost the same thing.

When this happens, developers usually react by tweaking the system prompt:

"IMPORTANT: You MUST always call fetch_customer_order when the user mentions an order! Do NOT call search_database!"

This is the Prompt Engineering Trap.

The root cause of tool-calling failures is almost never the model’s reasoning capacity. It is broken, sloppy tool contracts.

In this article, I will explain why agent tool schemas fail in production, break down the architecture of Toolsmith—a developer tool we built to lint, score, and harden AI tool definitions—and share the algorithms and engineering decisions behind it.

1. Anatomizing the Broken Tool Contract

Let’s look at a schema typical of many agent tutorials:

{
  "name": "lookup_user",
  "description": "Used to find information about people in the company or app.",
  "parameters": {
    "type": "object",
    "properties": {
      "user": {
        "type": "string",
        "description": "The user"
      },
      "filter": {
        "description": "Optional search parameters"
      }
    }
  }
}

Why does an LLM fail when presented with this?

Passive, Vague Description: "Used to find information..." does not give the model an imperative trigger. When should it call this instead of a general search tool? Under what constraints?
Missing Imperative Verb: High-performing tool descriptions start with active verbs (Fetch, Query, Search, Validate). This matches the instruction fine-tuning patterns of modern frontier models.
Untyped Parameter: "filter" has no "type". Is it a JSON object? A SQL string? A date range?
No Enum Sets: If "status" or "mode" only supports three valid states, failing to declare an enum: ["active", "suspended", "archived"] guarantees model hallucinations.
Missing required Array: The model has to guess whether "user" is mandatory or optional.

Multiply this by 15 or 20 tools in an enterprise agent runtime, and the model's tool-selection surface degrades into chaotic ambiguity.

2. Toolsmith System Architecture

To solve this, we designed Toolsmith: a tool that performs deterministic 0–100 linting of JSON tool schemas and provides an automated hardening loop powered by any OpenAI-compatible LLM endpoint (Ollama, Groq, OpenRouter, OpenAI) or Gemini.

Here is the high-level architecture:

┌────────────────────────────────────────────────────────────────┐
│                        USER INPUT                              │
│  - Raw Tool Schemas (tools wrapper, bare array, single tool)   │
└───────────────────────────────┬────────────────────────────────┘
                                │
                                ▼
┌────────────────────────────────────────────────────────────────┐
│                     INPUT NORMALIZER                           │
│  - Detects shape & standardizes into NormalizedTool[]          │
└───────────────────────────────┬────────────────────────────────┘
                                │
                                ▼
┌────────────────────────────────────────────────────────────────┐
│               DETERMINISTIC LINTING ENGINE                     │
│  (Pure TypeScript, Zero-Latency, No API calls required)        │
│                                                                │
│  ├── Description Clarity (Imperative verbs, min/max length)    │
│  ├── Parameter Hygiene (Types, enums, required arrays)         │
│  ├── Naming Conventions (snake_case, length, prefixing)        │
│  ├── Sibling Tool Overlap (Jaccard distance, lexical matches)  │
│  └── Token Economics (Byte size, token density, context cost)  │
└───────┬───────────────────────┬────────────────────────┬───────┘
        │                       │                        │
        ▼                       ▼                        ▼
┌──────────────┐        ┌───────────────┐        ┌───────────────┐
│ 0-100 Score  │        │ Tool Category │        │ Tool          │
│ & Grade Dial │        │ Breakdown     │        │ Complexity    │
│ (SVG arc)    │        │ (Progress)    │        │ (L / M / H)   │
└───────┬──────┘        └───────┬───────┘        └───────┬───────┘
        │                       │                        │
        └───────────────────────┼────────────────────────┘
                                │
                                ▼
┌────────────────────────────────────────────────────────────────┐
│                SESSION HISTORY ENGINE (Recharts)               │
│  - Tracks score improvement trajectory across analysis runs    │
└───────────────────────────────┬────────────────────────────────┘
                                │
                                ▼
┌────────────────────────────────────────────────────────────────┐
│               AI CONTRACT HARDENING ENGINE                     │
│  - Injects typed schemas, imperative verbs, enum constraints   │
│  - Connects to: Local Ollama, Groq, OpenRouter, or OpenAI API  │
│  - Side-by-Side Diff Inspector with field-by-field reasoning   │
└────────────────────────────────────────────────────────────────┘

3. The Deterministic Rule Engine

A critical design requirement for Toolsmith was determinism. Developers do not want to wait 8 seconds or spend $0.05 on API calls just to check if their JSON schema has typos or missing types.

The core linter runs synchronously in milliseconds:

export function runLinter(tools: NormalizedTool[]): LintReport {
  const findings: Finding[] = [];

  // 1. Description audits
  for (const tool of tools) {
    findings.push(...checkDescriptionQuality(tool));
    findings.push(...checkParameterHygiene(tool));
    findings.push(...checkNamingConventions(tool));
  }

  // 2. Cross-tool sibling overlap detection
  findings.push(...checkToolOverlap(tools));

  // 3. Score calculation
  const totalScoreDeductions = findings.reduce(
    (acc, f) => acc + f.scoreImpact,
    0
  );
  const overallScore = Math.max(0, Math.min(100, 100 - totalScoreDeductions));

  return {
    tools,
    findings,
    overallScore,
    grade: computeGrade(overallScore),
    categories: computeCategoryScores(findings),
    toolCount: tools.length,
  };
}

Checking for Imperative Verbs

Models trigger tools most reliably when descriptions start with active, unambiguous command verbs. The linter enforces this check:

const IMPERATIVE_VERBS = new Set([
  "get", "fetch", "search", "query", "find", "retrieve", "list", "read",
  "create", "insert", "add", "post", "generate", "build", "make",
  "update", "modify", "set", "patch", "delete", "remove", "cancel",
  "send", "execute", "run", "calculate", "compute", "validate", "check"
]);

function checkDescriptionQuality(tool: NormalizedTool): Finding[] {
  const findings: Finding[] = [];
  const firstWord = tool.description.trim().split(/\s+/)[0]?.toLowerCase();

  if (!IMPERATIVE_VERBS.has(firstWord)) {
    findings.push({
      id: `${tool.name}-desc-verb`,
      category: "description",
      severity: "warn",
      toolName: tool.name,
      field: "description",
      message: `Description should begin with an active imperative verb (e.g., 'Fetch...', 'Search...'). Currently starts with: "${firstWord}".`,
      fix: `Rewrite the first sentence to start with an active imperative command verb.`,
      scoreImpact: 8,
    });
  }

  return findings;
}

4. Measuring Tool Complexity

Not all tools carry equal cognitive load for an LLM. A tool with 8 top-level parameters and 3 layers of nested objects is significantly more likely to trigger execution errors than a simple key-value tool.

To expose this risk, we implemented a recursive tree walker that computes tool complexity:

export function computeToolComplexity(tool?: NormalizedTool): ToolComplexityInfo {
  if (!tool?.parameters?.properties) {
    return { level: "Low", paramCount: 0, nestedCount: 0, maxDepth: 0, summary: "0 params" };
  }

  let paramCount = 0;
  let nestedCount = 0;
  let maxDepth = 1;

  function walk(props: Record<string, any>, depth: number) {
    if (depth > maxDepth) maxDepth = depth;
    const keys = Object.keys(props || {});
    paramCount += keys.length;

    for (const key of keys) {
      const field = props[key];
      if (field?.type === "object" || field?.properties) {
        nestedCount++;
        if (field.properties) walk(field.properties, depth + 1);
      }
    }
  }

  walk(tool.parameters.properties, 1);

  let level: "Low" | "Medium" | "High" = "Low";
  if (paramCount >= 7 || nestedCount >= 2 || maxDepth >= 3) {
    level = "High";
  } else if (paramCount >= 4 || nestedCount >= 1 || maxDepth >= 2) {
    level = "Medium";
  }

  return {
    level,
    paramCount,
    nestedCount,
    maxDepth,
    summary: `${paramCount} param${paramCount === 1 ? "" : "s"}, ${nestedCount} nested`,
  };
}

The UI visualizes this with high-contrast badges (Low in emerald, Medium in amber, High in rose) right alongside each audited tool.

5. Visualizing Iterative Improvement with Recharts

When developers iterate on their schemas—or run our AI hardening engine—they need to see if their contracts are getting measurably better.

We integrated Recharts to render a lightweight, zero-gradient trend line:

<ResponsiveContainer width="100%" height={140}>
  <LineChart data={history} margin={{ top: 10, right: 15, left: -25, bottom: 0 }}>
    <CartesianGrid strokeDasharray="3 3" stroke="#1e293b" vertical={false} />
    <XAxis dataKey="label" stroke="#64748b" tick={{ fontSize: 10 }} />
    <YAxis domain={[0, 100]} stroke="#64748b" tick={{ fontSize: 10 }} />
    <Tooltip content={<CustomTooltip />} />
    <Line
      type="monotone"
      dataKey="score"
      stroke="#818cf8"
      strokeWidth={2}
      dot={{ fill: "#6366f1", r: 4, strokeWidth: 1, stroke: "#1e1b4b" }}
      activeDot={{ r: 6, fill: "#a5b4fc" }}
    />
  </LineChart>
</ResponsiveContainer>

This immediately surfaces the delta between an unoptimized schema (e.g. 42/100, Grade F) and its hardened variant (e.g. 96/100, Grade A).

6. Flexible AI Hardening Without `.env` Hassles

Most developer tools force you to clone a repo, configure a .env file, and restart containers just to test an AI model.

In Toolsmith, the OpenAI-compatible client configuration lives entirely in browser state:

Local Ollama: Works out of the box (http://localhost:11434/v1, model llama3.1).
Groq Cloud: Blazing-fast hardening via llama-3.3-70b-versatile.
OpenRouter / OpenAI: Plug in your personal API key and model of choice.

When the target endpoint is localhost or 127.0.0.1, Toolsmith calls it directly from the user's browser, avoiding server container networking bridges. For remote endpoints, it uses a lightweight server proxy to bypass CORS restrictions safely.

7. Key Takeaways for Production AI Agent Developers

Building robust agents requires treating tool schemas with the same rigor we apply to OpenAPI specs and gRPC protobufs:

Write imperative, trigger-focused descriptions: Never say "Used for search". Say: "Search customer orders by date range or customer ID. Call when the user explicitly requests order status."
Always define explicit enum values: If a status is ["pending", "shipped", "delivered"], write it out. Never leave it as an unconstrained "string".
Guard against overlapping tool scopes: If two tools have similar parameter names and verbs, merge them into a single parameterized tool or sharply distinguish their triggers.
Audit schema token weight: Tool definitions are injected into every single prompt turn. Bloated tool schemas silently eat away your context window and inflate latency.

Code & more: https://www.dailybuild.xyz/project/252-toolsmith

Voice Arena: I made two AI voice agents fight live on Agora and let the internet judge

Harish Kotra (he/him) — Sat, 12 Sep 2026 18:45:31 +0000

TL;DR: Voice Arena puts two real Agora Conversational AI Engine agents in one RTC channel, gives them the same prompt, and lets you vote the winner. Red (VAD + barge-in) vs Blue (semantic end-of-speech + keyword-only interruption). Real get_turns() latency receipts, ELO leaderboard, shareable knockout cards. Built with agora-agents 2.8.1, FastAPI, and a neon zero-build frontend.

Why I built it

Text LMArenas are solved. Voice is where the pain lives: Who feels faster? Does VAD or semantic end-of-speech win a roast battle? Does generated filler actually hide LLM lag? I wanted a toy that answers those questions by showing them — same prompt, two tunings, live audio, crowd vote. If you've ever argued about interruption settings in a PR, this is your boxing ring.

How it works

Prompt ─┬─► Red session ──► RTC channel ─┐
        │     (VAD EoS, SoS interrupt)    ├─► your ears + your vote
        └─► Blue session ─► RTC channel ─┘
              (semantic EoS, keyword-only)

Server (app.py, FastAPI):

POST /api/battle — text undercard. Two OpenAI-Compatible completions (MODEL_A/MODEL_B), blind-shuffled left/right, latency timed server-side:

text, ms, usage = await _chat(base, api_key, model, style["system"], req.prompt, temp, max_tokens)

POST /api/arena/start — the main event. Real SDK sessions via arena_agent.py:

client = AgentClient(area="GLOBAL", app_id=..., app_certificate=...,
                     customer_id=..., customer_secret=...)
red = build_fighter(client, "red").create_session(
    channel=channel, agent_uid="100", remote_uids=["*"])
red.start()
red.think(prompt, on_speaking_action="append")  # inject without killing the greeting

GET /api/arena/turns — post-fight receipts: session.get_turns() returns real per-turn ASR/LLM/TTS timings. No vibes, just milliseconds.
POST /api/vote — ELO (k=24) persisted to data/leaderboard.json.

Fighters (arena_agent.build_fighter): the tuning is the content.

OpenAI(api_key=..., base_url=..., model=..., temperature=...,
       system_messages=[{"role": "system", "content": SYSTEM_RED}])
.with_stt(AresSTT())
.with_tts(MiniMaxTTS(model="speech-2.8-turbo", voice_id=...))  # managed creds, no keys
.with_turn_detection({"mode": "default",
                      "config": {"end_of_speech": {"mode": "vad"}}})  # red: twitchy
.with_interruption({"mode": "start_of_speech"})
.with_filler_words({"enabled": True, "content": {
    "mode": "generated",
    "generated_config": {"fallback_strategy": "static"},
    "static_config": {"phrases": ["Give me a sec.", ...]}}})
.with_labels({"arena": "voice-arena", "fighter": fighter})

Swap one block and the whole fight changes character. That's the demo.

Frontend (static, zero build step): prompt roulette, live typing with EQ bars,
latency badges (⚡ms), SAY-IT-LIVE stage speech via session.say(), voice voting via Web Speech API
("left wins"), confetti, canvas-rendered 1080×600 knockout card, 𝕏 intent link,
and ?battle=id deep links so every fight is shareable.

Three things I learned

think(append) is the showrunner. Injecting the prompt with on_speaking_action="append" lets greetings land instead of getting clobbered. interrupt feels chaotic on stage; append feels produced.
Generated filler words earn their keep. With filler_words.mode=generated, the 1–2s LLM gap stops reading as "dead bot" and starts reading as stage presence. A/B it in the arena and you'll hear it instantly.
EoS mode is a personality slider. VAD Red interrupts, jokes, steps on lines. Semantic Blue waits, lands cleaner closers. Same models, different souls — and the crowd splits exactly the way you'd predict. That's the viral argument baked in.

Run it yourself

cd voice-arena
pip install -r requirements.txt   # agora-agents==2.8.1 pinned
cp .env.example .env              # add Agora + LLM creds
uvicorn app:app --port 8000       # open http://localhost:8000
python arena_agent.py --channel arena-demo --prompt "Dump me like a pirate."

Try: same model both sides, different temps → then different EoS modes → then different voices. Each change is one env var. Each fight is one tweet.

What's next

Heckle button (session.interrupt() live), clip exporter from turn timestamps, third fighter round-robin, avatar video knockouts. PRs welcome — keep SDK pinned, keep .env out of git, keep the fights blind.

Code & more: https://www.dailybuild.xyz/project/251-voice-arena

I Scraped All 234 YC Summer 2026 Startups and Found 31 That Are Just a SKILL.md

Harish Kotra (he/him) — Fri, 11 Sep 2026 19:40:30 +0000

A repeatable agent workflow is not a company. It's a skill. Here's the open-source proof.

Every few months a new YC batch drops, and with it a wave of "AI startups" that are really just an agent doing a repeatable task. So I did the obvious thing: I scraped all 234 companies in the Summer 2026 batch, read every one-liner, and asked one question.

How many of these could have been a SKILL.md?

A SKILL.md is a markdown file that tells an AI agent how to do something. It's a procedure: the steps, the pitfalls, the verification. If a startup's core value is a repeatable workflow an agent can follow, with no hardware, no regulation, and no capital moat underneath, then it is, functionally, a skill.

The methodology

I pulled the batch from the YC API (12 pages, 234 unique companies), then classified each one into three buckets:

SKILL (31, ~13%): the core is a documented procedure an agent executes. No heavy moat.
PARTIAL (130, ~56%): an agent plus a real moat (data, vertical integration, infrastructure).
NO (73, ~31%): hardware, robotics, chips, pharma, insurance, defense, space, physical infrastructure, biology.

The full classification is in yc-s26-skill-analysis.csv in this repo.

The pattern

What separates the 31 from the rest is striking. Their one-liners read like procedure descriptions:

"Least cost routing system to reduce 70%+ token spend"
"AI engineer that debugs and fixes production incidents in realtime"
"AI agents for compliance at institutional trading firms"
"Autoresearch as a service"
"AI-native debt collection company"

There is no proprietary hardware, no regulated license, no physical presence, no capital moat. The product is the workflow, the prompt, and the tooling around it. That is exactly what a SKILL.md is.

So I rebuilt all 31 as real skills

Each one is now a self-contained, open-source SKILL.md with a Procedure, the
Pitfalls, and a Verification step. No company names, no trademarks, just the
capability. A few examples:

token-cost-router — route every LLM request to the cheapest model that can handle it
production-incident-debugger — debug and fix a live incident fast and safely
government-contract-bidder — find, qualify for, and respond to gov contracts
collections-agent — polite-but-firm collections on overdue accounts
hipaa-internal-tool-builder — vibe-code and deploy HIPAA-compliant internal tools

How to use them

Skills are portable. Copy the folder into whatever harness reads SKILL.md:

Claude Code: ~/.claude/skills/<skill>/
Cursor: ~/.cursor/skills/<skill>/ or .cursor/skills/ in a project
OpenCode: ~/.config/opencode/skill/<skill>/
Hermes: ~/.hermes/skills/<skill>/

Then just ask your agent to do the thing. The frontmatter description makes it auto-discoverable.

The takeaway

The next time you see a funding announcement for an "AI agent that does X," ask whether X is a repeatable procedure. If it is, you don't need a subscription. You need a skill.

And if you're building one of these, the moat is never the workflow. It's the data, the distribution, the integration, or the trust you earn. The workflow is the table stakes.

Code & more: https://www.dailybuild.xyz/project/250-yc-s26-skills

Building Prism: a 100%-offline 3D visualizer for your AI coding history

Harish Kotra (he/him) — Thu, 10 Sep 2026 17:35:11 +0000

How a stdlib-only Python server, a vendored three.js snapshot, and a strict "metadata-only" contract turned scattered agent session stores into an explorable city — with no network, no database, and no build step.

The problem: my agents work while I sleep, and I have no idea what they did

I run half a dozen AI coding harnesses — Claude Code, Codex, OpenCode,
Antigravity, Pi, Goose… Each one keeps its own session store, in its own
format, in its own corner of my home directory: JSONL files here, SQLite
databases there, proprietary blobs somewhere else. I had hundreds of sessions and zero answers to simple questions: which project eats my tokens? When do I actually work? What does each model cost me? What tools get used?

The existing "solutions" were dashboards that wanted my API keys, my prompts, my code — uploaded to somebody's cloud. That felt backwards. The data was already on my disk. What I needed was a reader, not another service.

So I built Prism: open a localhost page, fly through your coding life in 3D.

Every design decision follows from three constraints: offline (zero runtime
network), read-only (never touch the harness stores), metadata-only
(never read a prompt, reply, or file).

Architecture: boring on purpose

The whole backend is Python stdlib only — http.server, sqlite3,
argparse, unittest. No pip install step exists, which means the install script is basically "make a venv, download one JS file, run":

./install.sh          # venv + vendor three.js (SHA256-pinned) + serve
python3 server.py --export prism-data.json --no-browser   # headless dump

The one network call in the project's life happens at install time
(curl three.js r128, hash-verified), and CI enforces the invariant forever:

make offline-check    # grep web/ for remote URLs → fail the build if any

Frontend is vanilla JS with no bundler. Each lens is a single file exporting { create }, and create(payload) returns { group, desc, legend, camera, tick, raycast, dispose }. That tiny interface is why nine lenses were cheap to build: Cityscape, Galaxy, Constellation, Timeline, Energy, Calendar, Tools, Hours, Models.

The heart: a schema that says "no"

The most important file is core/schema.py. Every reader normalizes into one document, and the schema gate rejects it before serving if anything is off — NaN/Infinity, negatives, boolean-as-number, malformed nested lists. A drifting upstream store format degrades to status: unavailable in the UI, never a 500.

A session (trimmed from a real export):

{
  "harness": "claude_code", "project": "flybench",
  "model": "claude-sonnet-4-6", "startedAt": "2026-07-09T14:02:11+00:00",
  "durationSec": 412,
  "tokens": { "fresh": 12011, "output": 3402, "total": 15413 },
  "tools": [{ "name": "Bash", "calls": 41, "errors": 2 }],
  "costUsd": 0.31, "costKnown": true
}

Notice what's absent: no prompts, no replies, no code. That absence is load-bearing — it's what makes "read my agent history" feel safe.

Readers: the extension surface

Each harness is one ~100-line file. The OpenCode reader shows the whole
pattern — including the discipline that matters most:

con = C.sqlite_ro(db)   # file:path?mode=ro + 500ms busy_timeout, ALWAYS
try:
    cur.execute("SELECT id, name, worktree FROM project")  # metadata tables
    # …never the message/part tables. Those hold conversation content.
finally:
    con.close()

Three helpers do the heavy lifting across all 28 readers: iter_jsonl
(skips malformed lines), sqlite_ro (read-only open), and
finalize_reader_payload (newest-first sort, 500-session cap, totals, meta).

Adding a harness is a 30-line file plus one fixture test — several Tier-2
readers (Cursor, Zed, Ollama…) sit stubbed in readers/ waiting for someone with the app installed to map the real layout.

Rendering 500+ sessions at 60fps with 2017-era three.js

The trick is old and unglamorous: InstancedMesh. Cityscape draws every session building in a single draw call; per-frame updates only rewrite instance matrices and colors for the time-scrub reveal. Same for the timeline river (markers + spikes), the calendar wall, the treemap. r128 is pinned because it's the last version that loads as one global script with zero modules — no bundler, no import maps, works from file:// if needed.

The fun engineering was anti-blob work. Real histories clump: same-day
sessions stack exactly, daily-driver projects form complete graphs, one tool (exec_command, 12k calls) eats the scene. Each lens got a declutter rule: timeline fans collisions into per-harness lanes, constellation keeps only ≥2-shared-day edges capped at top-2 per star, tools are log-scaled, treemap areas stay linear so "what you see is what you pay."

My favorite lens technically is Cityscape's true-shape maps: each of the 10 cities is hand-charted polygon data — Manhattan's diagonal island, the Seine S-curve with Île de la Cité, Sydney's branching harbour — with districts rejection-sampled onto real land, clustered downtown. No downloaded tiles, no textures; water is a plane, streets are line segments, landmarks are primitives. A subway map, not a satellite photo — and instantly readable.

Lessons worth stealing

Constraints are features. "No network, no writes, no content" killed entire categories of bugs (auth, sync, PII leaks) before they were born — and it's the whole pitch.
A validation gate beats defensive rendering. One strict schema check means nine lenses never think about malformed data.
Stdlib-only is a superpower for tools like this. The install is 30 seconds because there's nothing to install.
Graceful degradation > coverage. 8 verified readers + 20 honest "unavailable" stubs beats 28 half-broken parsers.

Prism is MIT-licensed. If you run coding agents, your history is already on your disk — go look at it: ./install.sh.

Code & more: https://www.dailybuild.xyz/project/249-prism

DEV Community: Harish Kotra (he/him)

Building "The One-Sentence Torture Test": making LLM comparisons falsifiable

The problem with "which model is better?"

The premise

Architecture: one module, two consumers

Design decisions that mattered

Never trust the model to report on itself

Empty output is a failure everywhere

Reasoning content is never read

SSE over POST

Per-provider reasoning toggle

The bugs that taught me the most

1. The SSE stream produced nothing at all

2. verify.ts silently booted a second server

3. Rung 7 only enforced half its instruction

4. Empty output reported as a charset violation

5. Duplicate React keys

6. The one that matters: 49/49 green while everything was broken

Verification in three layers

Proving multi-provider actually works

What the torture test actually reveals

If you want to build something like this

Try it

Building The Confidence Curve

The result

The design constraints that shaped everything

1. Grading happens in code, or it does not happen

2. reasoning_content never leaves the provider boundary

3. The crossover must be computed, never declared

4. Every call is written to disk before it is shown

Architecture

Rendering: making a 20-bar table legible

The bug that taught me the most

Building the surface live

The port collision that nearly produced a false result

What is verified, and what is not

What I would add next

Two Models, One Board: Building "Model vs Model"

Architecture

The core decision: the server owns the board

The retry/forfeit state machine

Gravity as a testable rule

The reasoning-token trap

SSE over POST

The three.js rendering

Testing without an API key

The bugs I actually hit

Playing any two models: the player library

What I'd build next

Caveat

Automating LLM A/B Testing: How to Build a Cross-Model Evaluation Harness Programmatically

The problem with ad hoc model testing

What Featherless contributes to the system

What we are building

How the async fan-out works

How candidate outputs are collected and scored

How the dataset is structured

A real example run

Why the judge model is separate

How the backend handles failures

How concurrency visibility helps

Why the storage layer is local

What to extend next

I made two LLMs draw the same cat. The constraint is the whole point.

Architecture

Bug 1: models typo the key, and I was deleting their drawings

Bug 2: the reveal stalled at 38% and never moved

Bug 3: reasoning models return nothing at all, and max_tokens is why

Bug 4: the loading state said "nothing is happening"

Rendering: from d attribute to tilted plane

Verification: no mocks, by design

Two things the suite taught me about my own claims

The finding I did not expect: reasoning kills the escalation

What I would tell you before you build something like this

Try it

I measured how deterministic LLMs actually are, prompt by prompt. The results broke my hypothesis.

The design

The prompt spectrum

The thing that makes or breaks the experiment: nonces

What the data actually showed

2. `verify.ts` silently booted a second server

2. `reasoning_content` never leaves the provider boundary

Bug 3: reasoning models return nothing at all, and `max_tokens` is why

Rendering: from `d` attribute to tilted plane

2. `EventSource` can't POST

`npm run verify`

`npm run ui-check` — and a measurement bug worth reading

`npm run mock`

6. Flexible AI Hardening Without `.env` Hassles