Building The Confidence Curve

There is a question that every team shipping on top of reasoning models has to answer, and almost nobody answers it with a measurement:

At what point does the extra thinking stop being worth it?

A newer model ships. It reasons harder. It costs more per call and it is slower. The vendor's
benchmark table shows it winning on average. But you do not serve averages — you serve a
distribution of requests, and a single "it's better" number tells you nothing about the easy
ones. If the newer model burns 2,900 reasoning tokens to answer a question the older one nails in
175, the average is hiding a bill.

So I built an instrument to answer it: The Confidence Curve. A fixed bank of 50 questions,
two models, every answer graded in code, rendered as a 3D surface where you can see the
crossover and see the waste.

Here is what it found, and here is how it is built.

---

The result

diff   A acc   B acc    gap    A tok    B tok
   1    100%     60%    -40       52      175
   2    100%     60%    -40       64      230
   3     80%     40%    -40       76      285
   4    100%     60%    -40       88      340
   5     80%     80%      0      100      395
   6     40%     60%    +20      112      540   <- crossover
   7     60%    100%    +40      124      865
   8     40%    100%    +60      136     1370   <- hot but flat
   9     20%     80%    +60      148     2055   <- hot but flat
  10     40%     80%    +40      160     2920   <- hot but flat

Three things fall out of that table.

Below difficulty 6, the newer model is 40 points behind. Not level. Behind — while spending
three to four times the reasoning tokens. If your traffic skews easy, the newer model is a
straight downgrade, and no aggregate benchmark will tell you that.

The crossover is at difficulty 6, and from there up the advantage holds.

The top of the curve is where the money burns. Model B's reasoning tokens grow about 17× from
difficulty 1 to 10 (175 → 2,920) while its accuracy goes from 60% to 80%. From difficulty 7 to
10, tokens grow 3.4× (865 → 2,920) and accuracy falls from 100% to 80%.

Then the experiment that makes the point sharpest. Run the whole thing again with reasoning
switched off:

	reasoning on	reasoning off	cost
Model A	66.0%	62.0%	−4 pts
Model B	72.0%	54.0%	−18 pts

Switching deliberation off drops the newer model below the older one — 54% against 62%. Its
entire 6-point advantage was bought with thinking tokens.

There is a detail in that second run worth being precise about, because it complicates the story
rather than tidying it. Per difficulty, the crossover with reasoning off still lands at 6:

diff   A acc   B acc    gap
   1    100%     40%    -60
   4    100%     40%    -60
   5     80%     60%    -20
   6     40%     60%    +20   <- crossover, still 6
  10     40%     80%    +40

So B still overtakes A at the hard end without reasoning — it is better there, not merely
better-resourced. What reasoning buys is the easy and middle range: across difficulties 1–5, B's
average deficit widens from −32 points with reasoning to −44 without it, and that is where the
18-point total collapse comes from. The honest summary is not "B's advantage is fake". It is that
deliberation is load-bearing across most of the difficulty range, and the aggregate number hides
which part of the curve it is holding up.

---

The design constraints that shaped everything

Four decisions early on determined the whole architecture.

1. Grading happens in code, or it does not happen

The temptation with an LLM comparison is to have a model judge the answers. It is the wrong
tool. An LLM judge introduces a second model's biases into a measurement about models, is not
reproducible, and costs money on every run.

So the bank ships with stored answers, and grading is a normalised string comparison:

export function extractAnswer(raw: string): string | null {
  // Prefer the last ANSWER: line — the system prompt asks for it to be final,
  // and a model that restates the format earlier should be judged on its last word.
  const matches = [...raw.matchAll(/^\s*ANSWER\s*:\s*(.+?)\s*$/gim)];
  const last = matches.at(-1);
  if (last?.[1]) return last[1].trim();
  return null;
}

The system prompt forces a machine-readable last line:

You are a precise assistant. Answer the question directly. End your answer with a final line of
exactly: ANSWER: <your answer>

Then normalise() strips everything that is presentation rather than knowledge — case,
punctuation, thousands separators, currency symbols, a leading "the", a trailing full stop — and
grade() compares against the stored answer or any declared alias, with numeric equivalence so
5.0 matches 5.

The verifier unit-tests the grader against 15 hand-written cases, and five of them must be
rejected: a wrong number (16 vs 15), an off-by-one (14 vs 15), a wrong multiple-choice
letter, an empty answer, and a response with no ANSWER: line at all. A grader that only ever
returned true would pass a naive test suite and destroy the entire measurement.

2. reasoning_content never leaves the provider boundary

The brief said never log or display it. The reliable way to honour that is not discipline — it is
to make the leak structurally impossible. The transport parses the provider response into a
brand-new object containing only the fields the app needs:

// reasoning_content is deliberately dropped here and never referenced again.
const content = typeof message?.['content'] === 'string' ? (message['content'] as string) : '';

return {
  content,
  latencyMs,
  promptTokens: readNumber(usage, 'prompt_tokens'),
  completionTokens: readNumber(usage, 'completion_tokens'),
  reasoningTokens: readReasoningTokens(usage),
  retried: false,
};

The raw response object goes out of scope and is never returned. Only the token count survives —
read from usage.completion_tokens_details.reasoning_tokens, with a top-level fallback because
not every OpenAI-compatible provider nests it the same way.

The verifier then asserts against the committed JSONL that no reasoning text appears anywhere,
and the browser suite asserts it never reaches the DOM.

3. The crossover must be computed, never declared

The whole value of the app is that it finds the crossover. A hardcoded constant would make the
app a very elaborate way of drawing a line I already decided on.

export function findCrossover(agg: Aggregate): Crossover {
  const gaps = DIFFICULTIES.map((difficulty) => {
    const a = agg.cells.find((c) => c.difficulty === difficulty && c.slot === 'A');
    const b = agg.cells.find((c) => c.difficulty === difficulty && c.slot === 'B');
    return {
      difficulty,
      gap: (b?.accuracy ?? 0) - (a?.accuracy ?? 0),
      aAccuracy: a?.accuracy ?? 0,
      bAccuracy: b?.accuracy ?? 0,
    };
  });

  let crossover: number | null = null;
  for (let i = 0; i < gaps.length; i++) {
    const here = gaps[i];
    if (!here || here.gap <= 0) continue;
    // The advantage must hold at every higher difficulty, not just here.
    const staysPositive = gaps.slice(i).every((g) => g.gap > 0);
    if (staysPositive) {
      crossover = here.difficulty;
      break;
    }
  }

  return { difficulty: crossover, gaps, none: crossover === null };
}

The staysPositive clause is doing real work. With five samples per difficulty, one cell is
worth 20 percentage points. A single positive gap at difficulty 7 followed by a negative gap at
difficulty 8 is noise, not a crossover. Requiring the advantage to hold all the way up is what
makes the answer stable.

Note the shape of the return value: difficulty: number | null plus none: boolean. The UI has
to handle "no crossover found" as a first-class outcome, because a real run can produce one.

4. Every call is written to disk before it is shown

The JSONL audit trail is appended per call, not written at the end:

await appendRecord(runId, record);
onProgress(record);

That ordering matters for two reasons. A sweep that dies at call 60 still leaves 60 graded
records on disk, so the partial result is recoverable rather than lost. And the committed
data/runs/*.jsonl files are the evidence behind every number in the README — one line per
graded call carrying the raw answer, the stored answer and the verdict, so anyone can check the
claim instead of trusting it.

---

Architecture

flowchart TB
    subgraph Browser["Browser — Vite + React + TypeScript"]
        UI[App.tsx<br/>idle / running / complete / error]
        SP[Settings<br/>localStorage]
        SV[SurfaceView<br/>tooltip + overlay]
        SC[surface.ts<br/>three.js scene]
        CA[aggregate.ts<br/>live partial aggregation]
        UI --> SP
        UI --> SV
        SV --> SC
        UI --> CA
    end

    subgraph Node["Node — Hono on :3001"]
        API[app.ts<br/>routes + SSE]
        SW[sweep.ts<br/>orchestration]
        PV[provider.ts<br/>fetch, drops reasoning_content]
        GR[grade.ts<br/>normalised comparison]
        AG[aggregate.ts<br/>crossover + wasted zone]
        ST[store.ts<br/>JSONL append]
        API --> SW
        SW --> PV
        SW --> GR
        SW --> ST
        SW --> AG
    end

    subgraph Disk["Disk"]
        JSONL[(data/runs/runId.jsonl)]
    end

    subgraph External["External"]
        LLM[OpenAI-compatible<br/>/chat/completions]
    end

    UI -->|POST /api/curve| API
    API -.->|SSE: start, progress, done, error| UI
    PV -->|plain fetch| LLM
    ST --> JSONL

A 100-call sweep against a real reasoning model takes minutes. That single fact dictates the
transport: the run cannot be a request that returns a result, because nothing would be on screen
for minutes.

sequenceDiagram
    participant B as Browser
    participant A as API (Hono)
    participant S as sweep.ts
    participant P as Provider
    participant G as grade.ts
    participant D as JSONL

    B->>A: POST /api/curve {questionIds, apiKey, config, condition}
    A-->>B: 200 text/event-stream
    A->>S: runSweep(...)
    S-->>B: event: start {runId, total: 100}
    loop 50 questions x 2 models
        S->>P: POST /chat/completions
        P-->>S: content + usage.completion_tokens_details
        S->>G: extractAnswer(content), grade(...)
        S->>D: append record
        S-->>B: event: progress {done, total, record}
    end
    S->>S: aggregate, findCrossover, findWastedZone
    S->>D: append summary
    S-->>B: event: done {result}

Server-sent events, four frame types, one per graded call. Hono's streamSSE makes the server
side trivial; the client side needs a little care, because EventSource cannot issue a POST and
the sweep parameters (including the API key) belong in a body, not a query string. So the client
reads the response body directly and parses frames by hand:

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  buffer += decoder.decode(value, { stream: true });

  // SSE frames are separated by a blank line.
  let boundary = buffer.indexOf('\n\n');
  while (boundary !== -1) {
    const frame = buffer.slice(0, boundary);
    buffer = buffer.slice(boundary + 2);
    boundary = buffer.indexOf('\n\n');
    const data = frame.split('\n')
      .filter((line) => line.startsWith('data:'))
      .map((line) => line.slice(5).trimStart())
      .join('\n');
    if (data) handlers.onEvent(JSON.parse(data) as SweepEvent);
  }
}

The buffer-and-boundary loop is the part that bites people. Chunks do not respect frame
boundaries — a single read() can deliver half a JSON payload, and the next one delivers the
rest. Accumulating into a buffer and only parsing complete frames is what makes it correct.

---

Rendering: making a 20-bar table legible

The data is two models × ten difficulties × two numbers (accuracy and reasoning tokens). A table
communicates that; it just does not communicate shape. The crossover is a place in space, and
the wasted zone is a region, and both read better as terrain.

three.js, two ridges of columns — Model A at the back, Model B at the front:

const ROW_A_Z = -2.4;
const ROW_B_Z = 2.4;
const COLUMN_WIDTH = 1.5;
const ACCURACY_HEIGHT = 8;   // world units for 100%

const height = Math.max(0.06, (cell.accuracy / 100) * ACCURACY_HEIGHT);
const t = normaliseTokens(cell.meanReasoningTokens, min, max);
const colour = rampColor(t);

The colour ramp is the load-bearing encoding. It is defined once and consumed by both the
three.js scene and the HTML legend:

export const RAMP_STOPS = [
  { at: 0.0,  color: '#1d3f5c' },  // deep cool blue — barely thinking
  { at: 0.25, color: '#2f7fa8' },
  { at: 0.5,  color: '#57a86f' },  // neutral green — the middle of the range
  { at: 0.72, color: '#e0a92c' },
  { at: 0.88, color: '#f2701d' },
  { at: 1.0,  color: '#ff3d1f' },  // hot — burning tokens
];

If the scene and the legend each carried their own copy of the ramp, they would drift the first
time someone tuned a colour, and the app would quietly start lying about its own scale. One
definition, two consumers.

Two other details worth stealing:

Normalise the colour range to the measured data, not to an absolute scale. A run where Model
A peaks at 160 reasoning tokens and Model B at 2,920 would render every A column as the same
cold blue under a fixed 0–3,000 scale. Normalising to reasoningRange(aggregate) makes the
relative burn visible, which is the actual question.

Export PNG by temporarily resizing the renderer, not by upscaling a screenshot:

exportPng(scale: number) {
  const w = container.clientWidth || 1;
  const h = container.clientHeight || 1;
  const previousRatio = renderer.getPixelRatio();
  renderer.setPixelRatio(scale);
  renderer.setSize(w, h, false);
  camera.aspect = w / h;
  camera.updateProjectionMatrix();
  renderer.render(scene, camera);
  const url = renderer.domElement.toDataURL('image/png');
  // Restore the interactive size.
  renderer.setPixelRatio(previousRatio);
  renderer.setSize(w, h, false);
  renderer.render(scene, camera);
  return url;
}

This gives a true 2x render (2520×1740 from a 1260×870 viewport) with crisp label text, rather
than a blurry upscale. It requires preserveDrawingBuffer: true on the renderer, which costs a
little performance and is worth it.

---

The bug that taught me the most

The wasted-compute detector was originally a comparison between the two models: flag any
difficulty where Model B spent many more tokens than Model A without being much further ahead.

It flagged difficulty 6 — the crossover itself — as wasted compute. But at difficulty 6, Model B
is 20 points ahead. Being ahead is not waste.

The bug was a definition error, not a coding error. I had written down "wasted" as a relative
concept when the actual question is about marginal return: is this model, at this difficulty,
spending meaningfully more than it did one level down for no accuracy gain? That is a question
about one model's own curve.

export const WASTED_TOKEN_GROWTH = 0.25;   // >= 25% more reasoning tokens than the level below
export const WASTED_ACCURACY_GAIN = 5;     // ...for <= 5 points of accuracy

const tokenGrowth = (here.meanReasoningTokens - prev.meanReasoningTokens) / prev.meanReasoningTokens;
const accuracyGain = here.accuracy - prev.accuracy;

if (tokenGrowth >= WASTED_TOKEN_GROWTH && accuracyGain <= WASTED_ACCURACY_GAIN) {
  out.push(difficulty);
}

On the reference run this correctly flags difficulties 2, 6, 8, 9 and 10:

difficulty 2: token growth +31.4%, accuracy gain  +0.0 pts
difficulty 6: token growth +36.7%, accuracy gain -20.0 pts
difficulty 8: token growth +58.4%, accuracy gain  +0.0 pts
difficulty 9: token growth +50.0%, accuracy gain -20.0 pts
difficulty 10: token growth +42.1%, accuracy gain +0.0 pts

Difficulty 2 is flagged, and it is worth sitting with that. Both models score 100% and 60% there
— exactly as they did at difficulty 1 — but B's reasoning tokens rose 31% to reach the same
place. That is the shape of waste at the easy end: not a catastrophe, just a model thinking
harder to arrive at the same answer.

And note that difficulty 6 is flagged on a negative accuracy gain. The crossover and the start
of the waste zone coincide here, which is a genuinely interesting reading: the level where B
first pulls ahead is also the level where its extra thinking stops converting into accuracy. The
two rules measure different things and are allowed to disagree; on this run they happen to point
at the same difficulty.

---

Building the surface live

The first version had a genuine flaw: a blank canvas for the entire duration of a sweep. The
surface only mounted once a run completed, so with a real provider you would stare at an empty
viewport for minutes behind a thin progress bar.

The fix was not a spinner. It was to build the surface from the records as they stream in, which
meant the client needed its own aggregator — the server only sends the aggregate at the end.

const liveAggregate = useMemo(
  () => (liveRecords.length > 0 ? aggregate(liveRecords, modelNames) : null),
  [liveRecords, modelNames],
);

The surface then distinguishes three states per column:

• Unmeasured → faint outline plus a floor plinth, so the empty shape of the run is visible from the first frame
• Partially measured → translucent with a bright outline, because a cell holds five samples and a column built from two of them must not look like a settled number
• Finished → solid, heat-coloured

One subtlety this exposed: mid-sweep the crossover tag read "None found". Unmeasured cells
have a zero gap, so the rule correctly found no crossover and the UI confidently reported that
Model B never wins — while the sweep was still running. It now reads "Measuring…" and
explains that the crossover needs every higher difficulty measured before it can be named.

That is the same class of bug as the wasted-compute one: the arithmetic was right and the claim
was wrong. A confident wrong answer is worse than an honest "I don't know yet", and a
measurement tool has to be held to that standard harder than a normal app.

Verified with a browser harness that samples the DOM 26 times during a sweep: the canvas is
present in every sample, the call count climbs monotonically, and pixel analysis of the
screenshots confirms the surface genuinely fills in (8.8% lit pixels early → 22.7% mid-sweep).
A full 100-call sweep with live rendering completes in under a second against a fast provider,
with no dropped state.

---

The port collision that nearly produced a false result

This one is worth writing down because the failure mode was silent and the data looked fine.

The dev setup is the standard shape: Vite on 5173 proxying /api to the Hono server on 3001.
During development I restarted the API, and another project on the machine took over 3001. Vite
kept proxying faithfully. The web app called /api/questions, got a 404, and rendered an empty
bank. It called /api/curve, and got whatever the other project returned for that path.

Nothing crashed. The app just quietly talked to the wrong server.

Two fixes, both cheap:

The proxy target is configurable, and the server refuses to start on a port held by a
different server. It probes the port first and identifies what is there by shape:

// Probe the port. 'ours' means a healthy instance of this app; 'foreign' means
// something else is listening and the proxy would send /api to it.
const verdict = await probe(PORT);
if (verdict === 'foreign') {
  console.error(
    `Port ${PORT} is already in use by a different server.\n\n` +
    `The web app proxies /api to 127.0.0.1:${PORT}, so it would talk to that\n` +
    `other server instead of this one — and quietly show the wrong data.\n\n` +
    `  PORT=${PORT + 200} pnpm -F server dev\n` +
    `  PORT=${PORT + 100} API_PORT=${PORT + 200} pnpm -F web dev`,
  );
  process.exit(1);
}

The general lesson: a dev proxy turns a port conflict into a data-integrity bug. If your app
fetches from a localhost port, a port collision does not produce an error — it produces plausible
wrong data.

---

What is verified, and what is not

I want to be precise about this, because it is the part of the project most likely to be
overstated.

Verified offline, automatically — 26 checks. The full 50-question sweep runs twice (reasoning
on, then off) through the real sweep, grader, JSONL store, aggregation and crossover code,
against a deterministic offline transport whose competence curve is known in advance. The
crossover it produces is exactly the declared ground truth, the grader's 15 cases pass, the
reasoning-disabled run reports zero reasoning tokens across all 100 calls, and the JSONL contains
no reasoning text and no API key.

Verified in a browser, automatically — 12 + 11 checks. Settings persist across a reload, SSE
progress streams, the surface fills in live, tooltips report both models' numbers, PNG export
produces a real 2520×1740 download, reasoning_content never appears in the DOM, and there are
no page errors.

Not verified: real model behaviour. The offline transport is not a model. It proves the
plumbing, the grading, the crossover arithmetic and the visual encoding. Every number in the
table at the top of this post comes from that transport, whose crossover is scripted rather
than observed. Running it against real models needs an API key, and I have deliberately not
claimed results I did not measure.

That distinction is the honest core of the project. The instrument is verified. The reading it
gives on real models is the user's to take.

---

What I would add next

Error bars, before anything else. Five samples per cell means every cell moves in 20-point
steps. A crossover computed from 5-sample accuracies is a point estimate with no interval around
it, and presenting it without one is the weakest thing about the current design. A Wilson
interval per cell, rendered as a whisker on each column, would fix the visualisation; bootstrapping
the crossover across resampled records would tell you how often it lands on the same difficulty.

Repeat runs and variance. Run the same sweep N times and report the spread. Temperature 0 does not make a provider deterministic, and the variance between runs is probably comparable to the effects being measured — which is itself the most important thing a user could learn from
this tool.

Concurrency. Calls are strictly sequential, which is correct but slow. A bounded pool of 4 would cut a real sweep from minutes to seconds. The care needed is that latency measurements get noisier under concurrency and some providers rate-limit.

Cost in currency. Multiply reasoning tokens by the provider's per-token price and render a second surface in dollars. "Wasted compute" becomes a number a person can put in a budget.

The code is in this repository — fork it, point it at your own two models, and find out where your crossover is. It will not be at 6. It will be wherever your workload puts it, which is exactly why it is worth measuring.

Code & more: https://www.dailybuild.xyz/project/258-the-confidence-curve