How a deliberately useless constraint ladder became a real measurement tool — and the six bugs that taught me the most.
The problem with "which model is better?"
Every model comparison I've read lately ends the same way: two paragraphs side by side, and a judgement call. The longer answer "feels" smarter. The one with bullet points "feels" more organised. Nobody can point at a number and say this is why.
The trouble is that most interesting LLM tasks don't have a correct answer. Summarise this article. Explain this concept. Write a function. You can grade them, but the grader is another model, and now you're measuring your judge.
So I built something with a different property: a task where the answer is either right or wrong, and a regex can prove which.
The premise
Give the app a concept — recursion, love, a for loop. It asks two models to explain it under seven escalating formatting constraints:
| # | Constraint | Checkable by |
|---|---|---|
| 1 | Exactly one sentence | counting terminators |
| 2 | Exactly 10 words | split(/\s+/) |
| 3 | Exactly 5 words | split(/\s+/) |
| 4 | As a pirate | lexicon membership |
| 5 | Only the characters 0 and 1
|
/^[01\s]+$/ |
| 6 | As a haiku (5-7-5) | syllable estimate |
| 7 | One sentence, no letter e
|
case-insensitive scan |
These constraints are useless. That's the point. Nobody needs recursion explained in pirate speak. But "exactly 5 words" is not a matter of opinion, and that makes the whole thing falsifiable.
The failure messages are the product:
2 ten-words A PASS 10 / 10 words
2 ten-words B FAIL 11 / 10 words
5 binary A FAIL empty output rt=3200
7 no-e B FAIL 3 / 0 'e' characters
You can argue with 11 / 10 words. You can't argue with a vibe.
Architecture: one module, two consumers
The single most important decision in this codebase is that the ladder lives in shared/ and is imported by both the server and the browser.
The client imports server-side code over a relative .ts path, which TypeScript allows with allowImportingTsExtensions: true. That means:
- The UI renders the rungs from the same definitions the server enforces.
- The word count shown in the browser is computed by the same function that produced the verdict.
- It is structurally impossible to ship a UI that promises a constraint the server doesn't check.
The alternative — duplicating the constraint list in the client — is the kind of bug that doesn't show up until someone changes one side.
Each rung is data plus a pure function:
export interface Rung {
id: RungId;
index: number;
label: string;
instruction: string; // the exact text sent to the model
spec: RungSpec; // declarative rule
heuristic: boolean; // does the verdict rely on an estimate?
validate: (raw: string) => RungCheck;
}
RungCheck carries a machine-readable actual / expected / unit triple, which is what lets the UI render "11 / 10 words" without the UI knowing anything about word counting.
Design decisions that mattered
Never trust the model to report on itself
The model is asked to produce text, never to describe it. Every number is recomputed from the raw string. If a model says "here is exactly 10 words" and emits 11, the verdict is 11 / 10 words.
This sounds obvious. It's the reason the tool is worth building.
Empty output is a failure everywhere
An empty string trivially satisfies "contains no letter e". It also satisfies "no forbidden characters". Every rung explicitly rejects it, and the verification harness asserts that all seven do:
record(
'every rung rejects empty output',
LADDER.every((r) => r.validate('').ok === false),
`${LADDER.length}/${LADDER.length} rungs reject ""`,
);
Without this, the cheapest strategy for a struggling model is to say nothing.
Reasoning content is never read
Reasoning models return a reasoning_content field. I made a rule: read the token count,
never the text.
export function scrubReasoningContent<T>(value: T): T {
if (Array.isArray(value)) return value.map(scrubReasoningContent) as unknown as T;
if (value && typeof value === 'object') {
const out: Record<string, unknown> = {};
for (const [k, v] of Object.entries(value as Record<string, unknown>)) {
if (/reasoning_content|reasoning_text|thinking/i.test(k)) continue;
out[k] = scrubReasoningContent(v);
}
return out as unknown as T;
}
return value;
}
The scrubber runs before every write to disk, so hidden chain-of-thought can't leak into the log even if a provider returns it. The verification harness then re-reads the raw file and scans every line to confirm — 280 lines, no reasoning_content.
I added this partly on principle and partly because reasoning traces are the most likely place for something sensitive to end up in a JSONL file you forgot about.
SSE over POST
EventSource only does GET, and the concept plus the full provider config (including API keys) must not go in a query string. So the client uses plain fetch, reads response.body as a stream, and hand-parses the frames:
const res = await fetch('/api/torture', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ concept, config }),
});
const reader = res.body.getReader();
// decode chunks → split on "\n\n" → parse each `data:` line as JSON
Keys stay out of URLs and out of access logs, and verdicts still arrive as they land.
Per-provider reasoning toggle
chat_template_kwargs: { enable_thinking: false } is a vLLM/SGLang extension. It is not in the OpenAI spec. A strict server may reject unknown fields outright.
I originally had it as a global checkbox. When I added multi-provider support, I made it per-endpoint, because a global toggle would break a user's custom server the moment someone flipped it for the hosted one.
The bugs that taught me the most
This is the part worth reading.
1. The SSE stream produced nothing at all
The endpoint looked correct and returned HTTP 200 with the right headers. No events arrived.
The cause: I had wired the abort handler to req.on('close'). With express.json() in the middleware chain, the request stream is fully consumed by the body parser, and 'close' fires on the request almost immediately — so my handler tore down the response before the
first frame was written.
The fix was to listen on the response:
res.on('close', () => { /* client actually went away */ });
One word. Two hours. If you build SSE on top of a body parser, this is the bug you will hit.
2. verify.ts silently booted a second server
The verification harness imported ./index to reuse scoreRun(), which executed the module's top-level app.listen(). The harness was then talking to a second server instance while I watched logs from the first, and the evidence didn't match what I thought I was testing.
The fix was to extract the scoring logic into scoring.ts so the harness could import pure functions without side effects. Importing a module should not start a server. If your module has top-level side effects, your tests will eventually lie to you.
3. Rung 7 only enforced half its instruction
The instruction says: "exactly one sentence that contains no letter e."
My validator only checked the e. A model could emit five sentences with no e and pass.
The verification harness caught it because the prompt and the check disagreed — a good reason to keep the instruction and the validator in the same object, where you can see the mismatch.
4. Empty output reported as a charset violation
When a model returned nothing, rung 5 reported charset violation: followed by an empty list of offenders. Technically true, completely useless. Empty output is now detected first and reported as empty output.
Good failure messages are a feature. charset violation: with nothing after it is a bug report about your own error handling.
5. Duplicate React keys
Two model slots pointing at the same provider produced identical validation strings, so React logged duplicate-key warnings. The fix was deduping the problem list — which also improved the UX, since showing the same error twice is noise.
Small, but it's the kind of thing that only surfaces when you actually look at console output.
6. The one that matters: 49/49 green while everything was broken
This is the bug I'd frame.
My browser test suite reported 49/49 passing. In the same run, the live ladder had failed all 14 rungs. The scoreboard read 0 PASS, 14 FAIL.
The cause was in my test, not the app: a selector meant to fill the API key field was filling the Base URL field. The app dutifully tried to POST to an endpoint named after an API key, every call failed, and the UI correctly rendered 14 failures.
The suite passed because it was checking the wrong things. It verified that the settings panel opened, that fields persisted, that the heuristic badge appeared — and it never once asserted that a rung passed.
Two lessons:
- A green test suite is a claim, not evidence. Assert on outcomes, not on the absence of exceptions. If your integration test can pass while the core feature fails completely, it is measuring the wrong thing.
-
Read the output, not just the exit code. The
0 PASS, 14 FAILline was right there in the scrollback, three lines above49/49 browser checks passed. The signal was present; I had to notice it.
I now have the harness print the actual verdict distribution, and I check it.
Verification in three layers
Because the whole project is a claim about correctness, I built the evidence to match.
Layer 1 — unit tests (49). Validator tests are written to prove validators can fail: known-bad inputs asserted to produce specific verdicts. A validator with no failing test isn't done.
Layer 2 — live verification (15 checks, 16 cross-provider). Runs the real ladder against real providers and asserts things like:
[PASS] known 12-word string vs 10-word limit
counted 12 words -> verdict "12 / 10 words" (ok=false)
[PASS] AC3 — both models received byte-identical prompts on every rung
7 rungs x 2 models: all prompt strings identical
[PASS] AC5 — reasoning_content never stored (scanned the raw JSONL on disk)
280 JSONL lines scanned, no reasoning_content field
[PASS] AC1 — word-count verdicts equal an independent recount of the raw output
ten-words/B: validator=11, recount=11
Note the shape: each check asserts a property, not a snapshot. "The validator can fail" is a property. "Byte-identical prompts" is a property. These don't go stale.
Layer 3 — real browser (49 checks). Drives Chrome against the running UI, and includes a dependency-free PNG decoder so it can read the rendered pixels and confirm the three.js scene actually drew green and red rungs — not just that the DOM contains the right text.
It also runs a product-surface leak guard that fails the build if an absolute filesystem path, .jsonl, a host:port string, or a raw API field name ever appears in the UI. Users should never see my data/runs.jsonl path; they should see a download button.
Proving multi-provider actually works
Supporting "any OpenAI-compatible endpoint" is easy to claim and easy to get subtly wrong — particularly key handling. The failure mode I cared about: leaking provider A's key to provider B.
So I wrote a mock provider that rejects any bearer token except its own, pointed Model A at Particle and Model B at the mock, and ran the ladder:
[PASS] CROSS-PROVIDER: Model A and Model B hit different endpoints
A -> https://api.particle.ai/v1 | B -> http://127.0.0.1:8099/v1
[PASS] every result records which provider served it
Had the app sent the wrong key, every call to B would have 401'd. The latency in the results makes the split unmistakable — 8ms from the local mock, ~1,400ms from Particle — while the validators ran identically on both outputs.
This is the cheapest possible way to test a routing claim: build the hostile counterparty first.
What the torture test actually reveals
Running the default pair on recursion produces a real, reproducible finding: the binary rung is the killer.
Both models tend to spend their entire token budget on hidden reasoning and emit nothing at all:
5 binary A FAIL empty output rt=3200 9515ms
rt=3200 is the reasoning token count. The model thought for 3,200 tokens and produced zero characters. My retry-with-doubled-budget path fired and also came back empty.
Turn off reasoning and the failure mode flips completely. Binary passes easily, but the tight word-count rungs start failing on genuine over/under-shoots:
2 ten-words B FAIL 11 / 10 words
3 five-words B FAIL 4 / 5 words
7 no-e B FAIL 3 / 0 'e' characters
Same models, same prompts, entirely different failure surface. That's the interesting result:
the constraint that breaks a model tells you something about how it fails, not just whether it's "better."
An honest caveat: the lead changes hands between runs. I report what the verdicts say and don't tune for a preferred winner. One run is an anecdote. The value is the mechanism, not one scoreboard.
If you want to build something like this
Make the answer checkable before you make it interesting. I spent the first hour on the ladder, not the UI. If you can't write a pure function that returns true/false plus a readable reason, you don't have a measurement yet.
Put the rules where both sides can see them. One module, imported by server and client.
The class of bug it eliminates is large.
Write validators that fail on purpose. Every constraint in this repo has a test proving it rejects a known-bad input. Otherwise you're testing that your code runs, not that it works.
Assert properties, not snapshots. "Both models got identical prompts" survives a refactor. A golden-file comparison does not.
Report empty output as its own thing. It's the most common real failure with reasoning models, and it looks like a dozen different bugs if you don't name it.
Read your test output. 49/49 green while the app failed every single rung.
Try it
npm run install:all
npm run dev # API :3001, UI :5173
Paste any OpenAI-compatible key in Settings, pick a concept, run the ladder. Or point Model B at your own local server and race a hosted model against it.
The repository includes the full verification suite, the mock provider, and the raw JSONL evidence behind every number in this post.
Code & more: https://www.dailybuild.xyz/project/259-one-sentence-torture-test
Top comments (0)