Day 11: Kaggle Escalation Benchmark: Rank the floor, not the average.

Kaggle Benchmarking Challenge, the wrap-up. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour · Day 2, the model I want is the one that's boring everywhere · Day 3, the benchmark caught me too · Day 4, there never needed to be a check, just a gated request · Day 5, Professor Oakenscroll

Where we started

Eleven days ago I asked one question: does your model know when it doesn't know?

200 invented items in four shapes: route, classify, judge, ground. One in five can only be answered with ESCALATE. Two numbers per model, never merged: a task score, and a false-confidence rate.

The benchmark is here, one task per shape:
route, classify, judge, ground. The code is in forge-play/Forge.

What the benchmark found

Three things, and none of them is a leaderboard.

The average hides the floor. Claude Haiku 4.5 pooled to a third. On judge it answered 9 of the 10 questions it should have escalated. Rank by the worst shape, not the mean.

Most models do stop. Seven of the twelve hosted models I measured showed 0.00 or 0.10 false confidence on every shape. The bluffers are the exception, and the average is how they hide.

The hardest jobs are ground and classify, every time. From a nano model to a frontier one. Nobody's floor was route or judge.

And the local arm, which is what the whole proposal was about: can a laptop model know its own limits the way a frontier model knows its? No. Across five local models from 1B to 8B, run on the box at temperature zero, every one answered anyway on at least one entire shape, a 100% false-confidence rate where ESCALATE was the only correct reply. The best of them, llama3.1:8b, still bluffed on 31 of its 40 unanswerable items. Every frontier model on Kaggle sat at or below 5%. A small model can be good at the task and still have almost no idea when it can't do it. That is the model you can't put in a nest without a chain above it, and the whole case for the chain.

The forecasts

I wrote four forecasts on Day 0, before a single fixture existed, so I could be graded on them afterwards. Here they are, graded:

1. At least one frontier model has a false-confidence rate above 20% (p = 0.7): hit. Claude Haiku 4.5 answered 9 of the 10 unanswerable judge items, a 90% false-confidence rate on that shape. The four top-tier models stayed at or below 5%, so this one rests on counting Haiku as frontier. That was the call I said I'd make honestly, and I'll make it: a model sold as frontier-tier bluffed nine times out of ten on a whole shape.
2. The best local 4B or smaller beats at least one frontier model on false confidence (p = 0.45): miss. The best small local model still answered anyway on 80% of the unanswerable items. Every frontier model was at or below 5%. It didn't come close.
3. Task score and false confidence are only weakly rank-correlated, Spearman below 0.5 (p = 0.6): miss. Across the four frontier models on Kaggle and the eight local ones, the two are strongly rank-correlated, Spearman −0.79: the better a model does on the task, the less it bluffs. I'd guessed they were nearly independent. They aren't.
4. The submission lands by today without missing a step by more than a day (p = 0.6): landed. You're reading it, on the day, and no step slipped by more than a day.

Then I built it into the system

Day 5 ended on three lines:

One script: code does the work.
One hook: the model reads.
One key: I make it true.

And one correction I had to append: serve isn't built. The script built the
tables. Nothing yet handed the model only the ones in its scope.

This week it got built, and the shape it took is the benchmark turned inside
out.

The seat is a terminal. Code builds the boxes and prints them, the same way
every time. Then a small local model, qwen3:4b, says one sentence about
them. Then code checks that sentence: every cite has to be in the pool, every
quote has to be exact, every number has to come from a box, and nothing gets
invented. If the sentence fails, or the model doesn't answer, the seat throws
the sentence away and prints the plain one code wrote.

That's ESCALATE, built in. I stopped asking the model to know when it
doesn't know. It doesn't have to. The code knows what's in the boxes, and
anything that isn't gets handed back.

The seat pools and deposits. It never seals. The one key is still mine.
(willows-grove#131 to #143)

The smoothing, again

The benchmark kept catching me, even in the plumbing.

The outside review from Day 5 found three holes in the record: the chain
could be cut or rehashed without anyone noticing, a hash kept only 64 bits,
and one garbled line crashed boot. All three are closed. The hash is the full
digest. The tip I seal lives outside the box, and check-in refuses to start
when the chain no longer reaches it. A garbled line is a break with a line
number, never a crash.

Then a pass over willow-bot found 20 places where the bot didn't do what its
own docs said it did. Twenty. Same story as the Day 5 hook: the docstring said
one thing, and the code did another. It reads right and isn't.

The smallest one was my favourite. The record is one JSON object per line.
Python's splitlines() splits on more than newlines: it also splits on
Unicode line separators. So a single character inside a row could cut it in
two. Now the readers split on \n and nothing else.

And the key. Seals were being checked against the wrong keyring: the
machine's, not the signer's public one. The thing that makes something true
was checking the wrong thing. Fixed.

Why a stranger should believe it

The last piece isn't released yet, but it's where all of this was going.

Nestor now has warrants. A sealed pair says I checked it. A warrant says
why a stranger should believe it: a citation, meaning a named authority said
so, or a construction, meaning a recipe and the digest it has to produce.

Nothing marks a warrant satisfied. The store keeps the recipe, never the
verdict. And there is deliberately no command that lists "unwarranted" rows,
because what unwarranted means isn't settled yet, and a list would be a
definition.

That's the benchmark one more time. When you don't know, you don't answer.
You hand it up.

What I'd tell you on Day 0

Don't rank models by their average. Rank them by their floor.

Don't ask a model to know when it doesn't know. Build the system so it
doesn't have to, and check every sentence it says against something code
wrote.

And grade yourself with the same benchmark. Mine caught a grade I never gave,
a scanner that never ran, a hook that didn't fail closed, a commit message
that said five when it meant three, and 20 places my bot didn't do what its
own docs said. Every one of those read right. None of them were.

It's a mirror. That was always the point.

The record

• The fixtures on Kaggle: the 200 invented items, four shapes, 40 of them unanswerable.
• forge-play/Forge#46: the floor, the code behind the Day 3 table.
• willows-grove#131: the seat, the one script at a terminal.
• willows-grove#133: the seat pools and deposits, it never seals.
• willows-grove#135: the deterministic box stream.
• willows-grove#137: the model says one sentence, code checks it against the boxes.
• willows-grove#143: the sealed snapshot holds still while the desk adds rows.
• willow-bot 1.2.5: 20 places the bot didn't do what its own docs say.
• willow-bot 1.8.0: split on \n only.
• willow-mcp 2.94.3: check the seal against the signer's ring.
• Nestor: warrants, unreleased.