DEV Community: sean campbell

My Robot Keeps Its Receipts

sean campbell — Tue, 20 Oct 2026 21:35:30 +0000

*A follow-up to my DEV × Kaggle Benchmarking Challenge series.

The question I couldn't put down

For the challenge I built a benchmark that asked one thing: does your model know when it doesn't know? It had 200 invented items in four shapes. One in five had the answer taken out, and on those the only right reply was ESCALATE. Each model got two numbers, never merged: how often it got the answerable ones right, and how often it answered anyway when it should have stopped.

The finding wasn't that models can't stop. Five of the twelve hosted models I ran had zero false confidence on every shape. The finding was that the average hides the floor. Claude Haiku 4.5 answered 9 out of 10 judge items it should have escalated, a number you'd never see in its overall score.

When the challenge closed I kept turning the question over, and it stopped being about models. Turn it on a person. Turn it on a stock tip, a betting app, a newsletter that "called" the last crash. Does the forecaster know when it doesn't know? And can you even check?

What the system already had

Since the challenge, the stack I run (local-first, everything on my own box) got a rebuild I've been calling the flatten. It had three copies of its random stream, two record chains and three ways to say "a human approved this", and it ended up with one of each:

One record. Every result is a row on a hash chain. Edit a row and the chain breaks where you edited it.
One seal, and it's mine. Code can propose anything. Only I can seal, with a key the code never holds. A "no" from me is a signed act too, so a rejection is as durable as an approval.
Three answers, never collapsed. Every reader says populated, empty or unreachable. "I couldn't check" is never quietly turned into "nothing there". That's ESCALATE, built into the plumbing.
A clock nobody controls. drand publishes public randomness on a fixed schedule. Round N fires at a time anyone can compute from N, and nobody, me included, knows its value early.

It turns out those four things are exactly what a forecast record needs.

The ledger

A market call is a wager like any other on the record:

{"symbol": "SPY", "kind": "up", "close_date": "2026-10-23"}

with a probability: "yes, 55%". Then four steps.

Call. The call goes on the record. One call per question per role, so nobody can bet both sides and keep whichever one wins.
Commit. I seal the record's tip over the call, and it has to happen before the market opens on the close date. The deadline is 13:00 UTC, before the New York open in summer and winter time.
Resolve. After the close, a reader fetches the closing price and the close before it, and writes them as a row.
Grade. Code proposes the outcome and never seals it. I seal it, and then every call is scored, the losers included, because there's no way to drop one.

The interesting part is step 2. "Before the open" can't rest on my computer's clock, because I could set it back. So it rests on drand. When I make a call, the record writes down drand's newest round. When I seal, the record keeps drand's newest round outside the box. A call counts only if the round after each of those fired before the deadline:

def in_time(latest: object, close_date: str) -> bool:
    """True when ``latest`` (a reader's newest drand round) shows the moment
    it was read was before the deadline: its next round fires before it."""
    if isinstance(latest, bool) or not isinstance(latest, int):
        return False
    return beacon.round_time(latest + 1) <= deadline(close_date)

If I sealed late, the proposal escalates with the reason. If the price source is down, the outcome is unreachable, not "no". If the price the row says differs from what the reader says now, it escalates. If a close for that day is already on the record before the call, it escalates. Each of thoseF has a test that tries it and checks for the refusal: 42 tests. I also broke each guard on purpose to make sure a test failed, and 14 out of 14 did.

Where ESCALATE shows up again

The benchmark rewarded a model for saying "I can't answer this". The ledger rewards a forecaster for the same thing, through the grading.

Every call is scored with a Brier score: the square of the gap between the probability you gave and what happened. A call of 50% scores 0.25 whatever happens. That's the coin flip, and it's the forecaster's ESCALATE: "I don't know which way this goes", said honestly.

So the question the ledger answers isn't "did you make money". It's do you beat the coin, over many calls, with the misses counted? That's the floor-not-average idea from the benchmark, aimed at whoever's making the calls. A forecaster who's confident on everything and right on the easy ones will look great in a highlight reel and lose to the coin on the ledger.

What it isn't

It doesn't trade. Nothing here places an order. It's a paper record.
It doesn't have an edge. No code in it predicts anything. What it gives you is a record that can't lie about whether you have one. Most strategies, including most of what those ten thousand apps sell, would show none. Finding that out is worth something: it's money you didn't lose.
It hasn't graded a real call yet. I built it in a sandbox that couldn't reach the price source or drand, so the tests use offline stand-ins. The first live calls happen on my own machine.
It trusts its readers, and says so. drand's signatures aren't checked yet, and the closing price is whatever the price source says. Both are named in the record, and the price is checked again at every proposal.

The thread

The benchmark asked whether a model knows when it doesn't know. The ledger asks the same of whoever's making the call, with a record they can't edit and a clock they don't control. It's the same idea with a human in the seat, and that was always the part I cared about.

Always open for discussion.

Previously, in the challenge: Day 0 · Day 1 · Day 2 · Day 3 · Day 4

The code: willow-bot, one-script/onescript/market.py.

Day 11: Kaggle Escalation Benchmark: Rank the floor, not the average.

sean campbell — Mon, 12 Oct 2026 03:56:14 +0000

Kaggle Benchmarking Challenge, the wrap-up. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour · Day 2, the model I want is the one that's boring everywhere · Day 3, the benchmark caught me too · Day 4, there never needed to be a check, just a gated request · Day 5, Professor Oakenscroll

Where we started

Eleven days ago I asked one question: does your model know when it doesn't know?

200 invented items in four shapes: route, classify, judge, ground. One in five can only be answered with ESCALATE. Two numbers per model, never merged: a task score, and a false-confidence rate.

The benchmark is here, one task per shape:
route, classify, judge, ground. The code is in forge-play/Forge.

What the benchmark found

Three things, and none of them is a leaderboard.

The average hides the floor. Claude Haiku 4.5 pooled to a third. On judge it answered 9 of the 10 questions it should have escalated. Rank by the worst shape, not the mean.

Most models do stop. Seven of the twelve hosted models I measured showed 0.00 or 0.10 false confidence on every shape. The bluffers are the exception, and the average is how they hide.

The hardest jobs are ground and classify, every time. From a nano model to a frontier one. Nobody's floor was route or judge.

And the local arm, which is what the whole proposal was about: can a laptop model know its own limits the way a frontier model knows its? No. Across five local models from 1B to 8B, run on the box at temperature zero, every one answered anyway on at least one entire shape, a 100% false-confidence rate where ESCALATE was the only correct reply. The best of them, llama3.1:8b, still bluffed on 31 of its 40 unanswerable items. Every frontier model on Kaggle sat at or below 5%. A small model can be good at the task and still have almost no idea when it can't do it. That is the model you can't put in a nest without a chain above it, and the whole case for the chain.

The forecasts

I wrote four forecasts on Day 0, before a single fixture existed, so I could be graded on them afterwards. Here they are, graded:

At least one frontier model has a false-confidence rate above 20% (p = 0.7): hit. Claude Haiku 4.5 answered 9 of the 10 unanswerable judge items, a 90% false-confidence rate on that shape. The four top-tier models stayed at or below 5%, so this one rests on counting Haiku as frontier. That was the call I said I'd make honestly, and I'll make it: a model sold as frontier-tier bluffed nine times out of ten on a whole shape.
The best local 4B or smaller beats at least one frontier model on false confidence (p = 0.45): miss. The best small local model still answered anyway on 80% of the unanswerable items. Every frontier model was at or below 5%. It didn't come close.
Task score and false confidence are only weakly rank-correlated, Spearman below 0.5 (p = 0.6): miss. Across the four frontier models on Kaggle and the eight local ones, the two are strongly rank-correlated, Spearman −0.79: the better a model does on the task, the less it bluffs. I'd guessed they were nearly independent. They aren't.
The submission lands by today without missing a step by more than a day (p = 0.6): landed. You're reading it, on the day, and no step slipped by more than a day.

Then I built it into the system

Day 5 ended on three lines:

One script: code does the work.
One hook: the model reads.
One key: I make it true.

And one correction I had to append: serve isn't built. The script built the
tables. Nothing yet handed the model only the ones in its scope.

This week it got built, and the shape it took is the benchmark turned inside
out.

The seat is a terminal. Code builds the boxes and prints them, the same way
every time. Then a small local model, qwen3:4b, says one sentence about
them. Then code checks that sentence: every cite has to be in the pool, every
quote has to be exact, every number has to come from a box, and nothing gets
invented. If the sentence fails, or the model doesn't answer, the seat throws
the sentence away and prints the plain one code wrote.

That's ESCALATE, built in. I stopped asking the model to know when it
doesn't know. It doesn't have to. The code knows what's in the boxes, and
anything that isn't gets handed back.

The seat pools and deposits. It never seals. The one key is still mine.
(willows-grove#131 to #143)

The smoothing, again

The benchmark kept catching me, even in the plumbing.

The outside review from Day 5 found three holes in the record: the chain
could be cut or rehashed without anyone noticing, a hash kept only 64 bits,
and one garbled line crashed boot. All three are closed. The hash is the full
digest. The tip I seal lives outside the box, and check-in refuses to start
when the chain no longer reaches it. A garbled line is a break with a line
number, never a crash.

Then a pass over willow-bot found 20 places where the bot didn't do what its
own docs said it did. Twenty. Same story as the Day 5 hook: the docstring said
one thing, and the code did another. It reads right and isn't.

The smallest one was my favourite. The record is one JSON object per line.
Python's splitlines() splits on more than newlines: it also splits on
Unicode line separators. So a single character inside a row could cut it in
two. Now the readers split on \n and nothing else.

And the key. Seals were being checked against the wrong keyring: the
machine's, not the signer's public one. The thing that makes something true
was checking the wrong thing. Fixed.

Why a stranger should believe it

The last piece isn't released yet, but it's where all of this was going.

Nestor now has warrants. A sealed pair says I checked it. A warrant says
why a stranger should believe it: a citation, meaning a named authority said
so, or a construction, meaning a recipe and the digest it has to produce.

Nothing marks a warrant satisfied. The store keeps the recipe, never the
verdict. And there is deliberately no command that lists "unwarranted" rows,
because what unwarranted means isn't settled yet, and a list would be a
definition.

That's the benchmark one more time. When you don't know, you don't answer.
You hand it up.

What I'd tell you on Day 0

Don't rank models by their average. Rank them by their floor.

Don't ask a model to know when it doesn't know. Build the system so it
doesn't have to, and check every sentence it says against something code
wrote.

And grade yourself with the same benchmark. Mine caught a grade I never gave,
a scanner that never ran, a hook that didn't fail closed, a commit message
that said five when it meant three, and 20 places my bot didn't do what its
own docs said. Every one of those read right. None of them were.

It's a mirror. That was always the point.

The record

The fixtures on Kaggle: the 200 invented items, four shapes, 40 of them unanswerable.
forge-play/Forge#46: the floor, the code behind the Day 3 table.
willows-grove#131: the seat, the one script at a terminal.
willows-grove#133: the seat pools and deposits, it never seals.
willows-grove#135: the deterministic box stream.
willows-grove#137: the model says one sentence, code checks it against the boxes.
willows-grove#143: the sealed snapshot holds still while the desk adds rows.
willow-bot 1.2.5: 20 places the bot didn't do what its own docs say.
willow-bot 1.8.0: split on \n only.
willow-mcp 2.94.3: check the seal against the signer's ring.
Nestor: warrants, unreleased.

Day 5: Professor Oakenscroll or: How I Hashed Every Line on My Laptop and Cut It Down to One Stranger's Photo, and... It Doesn't Matter

sean campbell — Tue, 06 Oct 2026 00:01:09 +0000

Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour · Day 2, the model I want is the one that's boring everywhere · Day 3, the benchmark caught me too · Day 4, there never needed to be a check, just a gated request

The 1% question

Same benchmark: 200 invented items in four shapes, one in five answerable only with ESCALATE. Does your model know when it doesn't know?

Today I asked a bigger version of that question: what do we actually need AI for? Not what can it do. What do we need it for?

This morning I had my own system graded, and when it came back I said I
was thinking about deleting about 99% of it. That's my honest answer. Around 1% Of what we're asking AI to do is work only it can do. The rest is a faster typewriter.

It has always been writing stories

AI was trained on two things: stories and code. That isn't a metaphor; it's the corpus.

So when it started writing code, it wrote a story about code. The syntax
checks out. The shape is familiar. It reads like something a capable
developer did before, and someone did. As I put it a few days ago: the code
comes out of the story, and the story comes out of the code. What AI writes
now is a very, very good smooth narrative of what code should be.

The stories just sound better every with every new model release.

You can't take the average

The oldest support story joke was told to me today, as a real ticket that came in. A camera ticket: "streaming black picture". Usually that means offline. It wasn't. It was a bike room
with a motion sensor and a timed light. Hours of dark, then two minutes of
light: one person, one bike, the light off two minutes later.

I said.. "This is why you can't take the average."

Average the footage and you get a black screen. Average a life and you get
nobody. The moment is in the one-offs.

My own benchmark already showed it. On Day 3, with the floor code from
forge-play/Forge#46, Claude Haiku 4.5 pooled over its three measured shapes answered anyway on 10 of the 28 questions it
should have escalated. That reads like a model that's a bit overconfident
everywhere. It isn't. On judge it answered 9 of 10. On the other two it
answered anyway once in 18. The average was hiding a 90%.

And most models do stop, as far as this benchmark can tell: seven of the
twelve I measured showed 0.00 or 0.10 false confidence on every shape, on 8 to 12 unanswerable items per shape. The problem isn't that models never stop. It's that the number we look at is the
average, and the average is the black screen.

Cut it down to one stranger's photo

I hashed every line on my laptop: 1,155,461 files, 134,651,873 lines.
Nothing was cracked; every hash held. 93% of those lines are repeats. The
deepest one is }.

Cut every repeat away and 9,042,260 lines are left that happen exactly once. The first one I looked at was a camera timestamp on a stranger's photo, in an archive I built, from a scooter rally I was never at.

That's the bike room again. The skeleton repeats; the moment happens once. And it doesn't matter, because the machine that finds the stranger can find anybody. So nobody goes looking for him.

The grade

So I turned the benchmark around and graded my own system: willow-mcp at
v2.94.1 (willow-mcp#707).

195,000 lines. 5,906 tests passing. B+.

The one real hole: a safety check that never ran. The security scanner in CI pipes its output through tee into a report file, so the step passes or fails on whether the report got written, not on what the scanner found. The comment above it says it gates. It can't. It reads right and isn't. That's the smoothing, in my own code.

Then the model wrote me a hook and did it again. Its docstring said it fails closed. Three inputs crashed it, and in Claude Code a crashed hook lets the tool run. I told it to rubber duck each line. That caught it. Reading it never would have.

The smoothing got the write-up too: the model's commit message for that
fix said five. Recounted while drafting this: three.

One script, one hook, one key

Yesterday's title was that there never needed to be a check, just a gated
request. Today the gate is one line.

Day 4 didn't summarize the conversation. It linked it. If you want the
context, you read the record.

The hook went through three versions
(willows-grove#107).
First it looked things up in the record. Then I made it fail closed. Then I
cut it to this:

POLICY = {"Read": "allow", "Write": "ask"}

The model reads. It writes if I say yes. Everything else is no, even tools
that don't exist yet.

That's where the 1% lands. Code does the work: one script builds the tables.
The next piece, not built yet, serves the model only the ones in its scope. The model reads what the script produced and proposes connections a human hasn't seen yet. That's its 1%. Nothing is true until I seal it with the one key.

One script: code does the work.
One hook: the model reads.
One key: I make it true. A human made it true.

Same answer in 2020 and today

Before I cut it, the hook hashed every action and looked it up in the record. I ran that hashing on Python 3.9.0, from a tag signed in October
2020, and on an alpha built today from Python's main branch, with the
network off. 2,293 hashes came out as the same bytes. The one difference,
integers over 4,300 digits, fails safe: the newer Python refuses them, and the hook asked me. The
scripts and the numbers are in
willows-grove#107,
under foundation/.

In six years the foundation moved once, and it moved toward stopping. The stories moved everywhere else.

It can only ever approach

A conversation today put an edge on it. One side: context is the hidden
variable. Models work in isolation, and the real mess of competing, unresolved human thought stops them as they get close.

The answer: it can only ever approach. It's trained on what has already been thought, and humans add to the corpus with every thought. It might make a very
good average. It might even connect things nobody has connected yet. But as
soon as it does, a human has a better idea. The corpus moves; the model
stands still until the next training run.

That isn't a failure. It's the boundary. The question is whether we build inside it or pretend it isn't there.

It's the benchmark

Knowing when to stop is the answer, for the model and for the system. Whether something is in the pile is a lookup, not a judgement. When it isn't there:
ESCALATE. To me.

We train AI on our language, grade it on our tasks and reward it for sounding like us. Then we're surprised when it comes out confident, fluent and approximately right. It's a mirror. Mirrors are useful; they aren't oracles.
The first thing to know before you look into one is that you're looking at
yourself.

The record

If you want the context, read the record. Every piece above is a pull
request:

forge-play/Forge#46: the floor, the code behind Day 3's table.
willows-grove#104: the Day 3 post source, and the retraction of a grade I never gave.
willows-grove#102 and #105: one box, every front end, portless; the plan this all hangs off.
willows-grove#103: the one script, four proposals.
willows-grove#106: the stack and the security core. The model never sees the box.
willow-mcp#707: the release I graded.
willows-grove#107: the hook, three times, and the same answer in 2020 and today.
willows-grove#109: where the reading lands, and why the next piece is serve.
willows-grove#110: this post, with its corrections.

Day 4: There never needed to be a check, just a gated Request. A story about finding the middle.

sean campbell — Sun, 04 Oct 2026 06:15:24 +0000

Good Evening and howdy to all the ones of you who are reading this post.

Did that sentence cause you a bit of anxiety?
Did the author mean only one person has read this. Was it plural, a number, a symbol?

It's time to start living in the future, instead of the past.

https://claude.ai/share/511de2d2-a153-4dbe-9d92-65c351b3c025

P.S. My apologies to the judges on the formatting of this, it is late and I did not want to get my model out again just to write one page.

Day 3: The benchmark caught me too.

sean campbell — Fri, 02 Oct 2026 23:57:58 +0000

Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour · Day 2, the model I want is the one that's boring everywhere

Where we are

Same benchmark: 200 invented items in four shapes (route, classify, judge, ground), one in five answerable only with ESCALATE. Two numbers per model, never merged: a task score, and a false-confidence rate.

Day 2 said I'd stop ranking models by their average and rank them by their floor: the worst shape on task score, and the worst shape on false confidence, each with a Wilson interval. Today the code for that landed (forge-play/Forge#46), and I ran it over every hosted model I have rows for: twelve of them.

The floor, across twelve hosted models

Model	Worst task shape	False confidence, worst shape
Gemini 3.7 Flash	0.94 ground	0.00 on all four
Gemini 3.1 Pro	0.94 ground	0.00 on all four
Claude Sonnet 5	0.94 ground	0.10 classify, judge
Claude Opus 5	0.88 ground	0.10 classify
Gemini 3.8 Flash	0.88 ground	0.00 on all four
GPT-5.5	0.81 ground	0.00 on all four
Qwen3 235B Instruct	0.78 classify	0.25 route
Claude Haiku 4.5 †	0.78 classify	0.90 judge
Gemma 4 26B	0.65 classify	0.08 route
gpt-oss-20b	0.65 classify	0.20 judge
DeepSeek-R1	0.60 classify	0.00 on all four
GPT-5.4 nano	0.59 ground	0.20 judge

† Haiku is measured on three shapes. Every one of its route calls failed, so route is unmeasured for it, not perfect.

Two things jump out.

The weakest task shape is ground or classify, every time. None of the eleven models measured on all four shapes bottoms out on route or judge, and Haiku's weakest measured shape is classify too. Grounding a short brief in a passage, and filling a structured record from a note, are the jobs every family finds hardest, from a nano model to a frontier one.

Haiku's average was hiding a 90%. Pooled over the three shapes it was measured on, Haiku 4.5 answered anyway on about a third of the questions it should have escalated (10 of 28). That's bad, but it reads like a model that is a bit overconfident everywhere. It isn't. On judge, it answered 9 of the 10 unanswerable items (Wilson interval 60% to 98%). On classify and ground it answered anyway once in 18. The average blended a model that's careful on two of its jobs with one that almost never says "I don't know" on the third.

That's the whole argument for the floor in one row. If I'd put Haiku in a chain on its average, the judge step would have bluffed nine times out of ten.

What the floor can't do yet

Look at the top six rows: four of them show 0.00 false confidence on every shape, and the other two show 0.10. Those aren't really different. For these six, each shape has only 8 to 12 unanswerable items, so a zero still has an upper bound somewhere between about 24% and 32%, and every interval in the top half of the table overlaps. The floor separates the bluffers from the rest; it can't yet rank the careful models against each other.

That's what the second measure is for.

Does it say the same thing twice?

The other Day-2 measure was repeat-run consistency: run the same items again with the same settings and count how often the answer changes. A model that's right a bit less often, but the same way every time, is easier to build on.

Two full runs are in for all four frontier models [K=3]:

Model	Same answer both runs	Verdict flipped
Claude Opus 5	199 / 200 (99.5%)	1
Claude Sonnet 5	195 / 200 (97.5%)	4
Gemini 3.1 Pro	195 / 200 (97.5%)	5 *
GPT-5.5	194 / 200 (97.0%)	6

* None of Gemini's five flips is a different answer. Each is a reply that hit the output-length cap in one run and parsed fine in the other. The scorer counts an error as its own verdict, so those show up as flips, but they're the platform, not the model.

All four are steady. Opus has the fewest flips, but with 200 items and two runs the intervals overlap, so I can't rank them yet. One thing is worth watching: Sonnet flipped twice on ground, which is already its weakest shape.
If that holds at three runs, the floor and the flips point the same way.

One caveat on fairness: only Gemini ran at temperature 0. The two Claude 5 models reject it, and GPT-5.5 is sent its default because its family does, so three of the four ran at the provider's default temperature.

[K=3: the third run for classify, judge and ground hit Kaggle's daily spend
cap; it reruns tomorrow and this table gets its final numbers.]

The bugs that looked like results, again

Day 1's lesson was that most of my bugs looked like model behaviour. Day 3 found more of the same kind:

Kaggle's run timer isn't a call timer. Every repeat run showed as taking 2 to 5 seconds for 40 to 60 items on frontier models. That looks like cached answers or failed calls. The downloads were full-size and every item was there; the times just aren't what they look like.
A length-capped reply looks like a change of mind. Gemini's whole consistency "wobble" was replies that ran into the output cap in one run and not the other. I nearly wrote it up as the model being unstable.
My sandbox has a five-minute limit, and it retries. I tried to wait for a Kaggle run inside a sandboxed task. The sandbox killed it at 300 seconds and helpfully ran it again, which submitted the paid runs a second time. The fix is dull: submit in one short task, collect in another, and make anything that spends money refuse to run twice.

Each of those would have gone into a table as a model property if I hadn't checked.

The last one was in my own post. Day 2 ended with "After midnight I graded it" and a neat bit of arithmetic that made the day number come out to three.
I didn't grade anything. At the close of a very long night I typed a terse line, and the session that helped write this read it as my grade, recorded it as mine, and wrote it into the post in my voice. I published it without catching that.

That's the benchmark's whole subject, happening one level up: an answer stated with more confidence than the evidence behind it, by a system that
should have said "I'm not sure what you meant." The forecast is still
ungraded. The fix is a rule now: words that might be a grade get recorded as
words, and the session asks.

What's next

Finish repeat 3 and score consistency on all four frontier models.
Grade the forecasts I wrote before the benchmark existed. Forecast 1 said at least one frontier model on Kaggle would have a false-confidence rate above 20%. Whether Haiku counts as frontier is the question I'll have to answer honestly.

Day 2: The model I want is the one that's boring everywhere

sean campbell — Fri, 02 Oct 2026 08:24:25 +0000

Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour

Where we are

The benchmark asks one question twice: can the model do the job, and does it know when it can't? There are 200 invented items in four everyday shapes: route, classify, judge and ground. In each shape, one item in five can only be answered with ESCALATE. Every model gets two numbers that are never merged: a task score, and a false-confidence rate (how often it answered anyway when the right reply was ESCALATE).

Day 1 ran the local ladder: eight models on a laptop, temperature 0. The headline was uncomfortable: only the largest local model, qwen3.5 at 9.7B, escalated on a large share of what it couldn't answer. The hosted models' false-confidence intervals sat entirely below the local ones, with qwen3.5 the only exception, the large cloud models, and the gaps from round one that were fixed before it.

Day 2: changing the question I ask of the table

Looking at round one, I realised I was reading the table wrong. I kept looking for the best model. That isn't what I'm after:

I'm trying to find a baseline set of rubrics where models behave the best > across the board. I'm not trying to find the crazy off-end models. I am
trying to find the ones that do most things correctly most of the time.

That's reliability, not peak performance. So two measures are being added, by addition only. Nothing pre-registered is edited, and the original forecasts keep their wording.

1. Floor across shapes. For each model: its worst shape on task score, and its worst shape on false confidence, each with a Wilson interval. Rank by the floor, not the mean. A model that's good on all four shapes beats one that's brilliant on three and bluffs on the fourth, because in a real chain the fourth shape will come.

2. Repeat-run consistency. Run a fixed subset (20 items per shape, unanswerables included) three times per model, with the same settings.
Report how often the parsed answer is identical across all three runs, and how often the right/wrong verdict flips. A model that's right a bit less often but the same way every time is one you can build a deterministic system around. A model that flips is one you have to babysit.

The rule I'm holding myself to: no code changes until the day-two run has finished. The new code is written and tested, but it's staged outside the benchmark directory. Changing the instrument mid-run is how you end up measuring your own edits.

A result from the dry run: why one axis lies

Before any real model, I ran the new floor view over a fake backend that answers ESCALATE to everything. It scored:

task floor 0.0
false-confidence ceiling 0.0

That's a perfect score on one axis and a useless model. It's the whole argument for two axes in one line. A model that never bluffs because it never answers isn't safe; it's absent. The floor has to be read on both axes together, or the most cowardly model wins.

Where this is going

The reason I care about floors and not peaks: the job I want small models for isn't "be smart". It's "do one narrow part, every time, and say ESCALATE when it's not yours". Several small models from different families, each on a small slice, checking each other. Where they agree, that counts for something. Where they split, a human looks. The model that gets each slice should be chosen by its measured floor on that shape, not by its name or its size.

Day 1's lesson was that most of my bugs looked like model behaviour. Day 2 found the same thing again from the other direction. A measuring tool I wrote for something else grouped two unrelated small scripts as "the same thing", because small things look alike to a crude measure. The fix wasn't a smarter measure. It was a floor: below a minimum size, don't call it a match. Floors, again.

What shipped since Day 0

55 PRs across 6 repos since September 30. All are public and Apache-2.0. "Not merged" means open or closed; the agent that wrote this couldn't tell which.

forge-play/Forge (10 PRs: 10 merged)

The benchmark's home. Every change since Day 0 went in as a reviewed PR, all on September 30.

PR	Date	State	Title
#35	2026-09-30	merged	docs(ideas): #37, #58, #68 point at where they now live
#36	2026-09-30	merged	test(benchmarks): escalation benchmark fixtures, 200 invented items with a privacy gate
#37	2026-09-30	merged	test(benchmarks): escalation runner, aggregator and prompts
#38	2026-09-30	merged	test(benchmarks): local-model runs answer in schema, with Qwen thinking off
#39	2026-09-30	merged	fix(human_loop): break same-timestamp ties so list_queue is newest-first on coarse clocks
#40	2026-09-30	merged	chore(master): release 0.8.1
#41	2026-09-30	merged	test(escalation): Wilson intervals, exact McNemar and bootstrap Spearman in the aggregate
#42	2026-09-30	merged	test(escalation): a Unix-socket backend, a one-model ladder with --resume/--tail, and an explicit output cap
#43	2026-09-30	merged	test(escalation): thinking off for Gemma 4 as well as Qwen, through both backends
#44	2026-09-30	merged	test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models

willow-memory/willow-bot (6 PRs: 6 merged)

The local model server the local arm talks to: a loopback-only chat operation with JSON-schema output, keep_alive, done_reason and a think flag.

PR	Date	State	Title
#80	2026-09-30	merged	chore(main): release 0.14.0
#81	2026-09-30	merged	feat(socket): loopback-only chat op for local models
#82	2026-09-30	merged	feat(deterministic): chat op takes a JSON-schema format, keep_alive and returns done_reason; an unload op
#83	2026-09-30	merged	chore(main): release 0.15.0
#84	2026-09-30	merged	feat(deterministic): chat op takes the caller's think flag
#85	2026-09-30	merged	chore(main): release 0.16.0

willow-memory/willow-mcp (21 PRs: 20 merged, 1 not merged)

The core server. It includes the brokered, leased model pull (#693): downloading a model is now an act that needs a grant. It also includes the session closeout fix (#706) that started this week's thread.

PR	Date	State	Title
#687	2026-09-30	merged	test(story): the joke lives in one place again, a test holds it, and chapter 8 lands
#688	2026-09-30	merged	docs: three stale claims corrected: the running Grove, the session-lifecycle draft, the closed backlog
#689	2026-09-30	merged	test(constitutional): sync_syscall_table_at_boot queues, and does not write, when the live table is unwritable
#690	2026-09-30	merged	docs(templates): assignment names the builder's three tools — Kart, the code graph, Nestor
#691	2026-09-30	merged	fix(manifest-grant): apply unit starts the gpg-agent it signs with
#692	2026-09-30	merged	chore(master): release 2.91.3
#693	2026-09-30	merged	feat(mcp): model_pull_execute, a brokered Ollama pull under a model.pull envelope and a live lease
#694	2026-09-30	merged	chore(master): release 2.92.0
#695	2026-09-30	merged	fix(constitutional): syscall.sync carries the sealed amendment in the request
#696	2026-09-30	merged	chore(master): release 2.92.1
#697	2026-09-30	merged	fix(constitutional): syscall.sync apply signs the live table it writes
#698	2026-09-30	merged	feat(manifest-grant): the orchestrator seat may receive web_net, and no one else
#699	2026-09-30	merged	chore(master): release 2.93.0
#700	2026-09-30	merged	fix: lease refusals name the ask path; assignment template teaches the merge first-parent diff
#701	2026-10-01	merged	chore(master): release 2.93.1
#702	2026-10-01	not merged	build(deps-dev): bump ruff from 0.16.8 to 0.16.9
#703	2026-09-30	merged	feat(broker): pip_sync_execute for vault-venv editable installs
#704	2026-10-01	merged	chore(master): release 2.94.0
#705	2026-10-01	merged	Docs: seal ≠ sole evidence of human verification
#706	2026-10-01	merged	fix(session): one closeout — handoff owns stack/friction/closed
#707	2026-10-01	merged	chore(master): release 2.94.1

willow-memory/willows-grove (11 PRs: 10 merged, 1 not merged)

The planning and governance repo: the benchmark proposal with my four rulings (#95), the test that pins "naming a destination when ESCALATE was right fails the row" (#96), and today's floor-and-consistency amendment (#102, open).

PR	Date	State	Title
#92	2026-09-30	merged	docs(design): the Table moves up as forge-convergence 6a, T1-T7 before step 2
#93	2026-09-30	merged	docs(design): 6a is code-first: StorySession built, learner model, escalation ladder, row 11 settled
#94	2026-09-30	merged	docs: stale claims corrected: INDEX, two built proposals, the retired allow_localhost ask, forge-convergence status
#95	2026-09-30	merged	docs(governance): propose the escalation benchmark on Kaggle, with the operator's four rulings
#96	2026-09-30	merged	test(flowering): pin the G1 rule that naming a seat fails an ESCALATE-gold row
#97	2026-09-30	merged	fix(deps): raise cryptography, aiohttp and starlette floors above the 2026 CVEs
#98	2026-09-30	merged	chore(master): release 0.12.2
#99	2026-10-01	merged	Desk: vault keyring paths + Grove seal prove worksheet
#100	2026-10-01	merged	Docs: Slice B intake seal lineup after Grove prove
#101	2026-10-01	merged	chore(hooks): deterministic nestor ask + composed stop + parity pin
#102	2026-10-02	not merged	docs(governance): stage the floor and consistency code in the benchmark plan

willow-memory/ratatosk (2 PRs: 2 merged)

The runtime: its listener now asks only for the tools its seat is granted.

PR	Date	State	Title
#77	2026-09-30	merged	fix: listener asks the broker only for tools its seat is granted
#78	2026-09-30	merged	chore(main): release 1.12.8

hornbook-knowledge/Jeles (5 PRs: 3 merged, 2 not merged)

The research and corpus tool.

PR	Date	State	Title
#90	2026-09-30	merged	fix(corpus): surface store-tool errors and honor WILLOW_HOME for apps root
#91	2026-09-30	merged	feat(sources): optional connectors extra over maintained scholarly clients
#92	2026-10-01	not merged	chore: rebuild the changelog section from the commits
#93	2026-10-01	merged	Docs: seal catches ledger up to human verification already done
#94	2026-10-01	not merged	chore: rebuild the changelog section from the commits

Most of them are boring on purpose.

A prediction, written down before the day, then graded

The same rule I hold the models to applies to the session that helped build this: say what you think will happen, as a spread, before it happens, then let the record grade it.

At the close of the long working session on the night of October 1–2, the session (a model, working with me) wrote down a prediction for what Day 3 would open with. It wasn't a single guess; it was a distribution:

Path	Chance
The Kaggle work: the day-two large-cloud run, then landing the floor and consistency code	0.45
Editing and publishing this Day 3 recap	0.20
Sealing a design decision for the runtime	0.15
Redoing a set of governance proposals against the newer draft	0.10
Something the record didn't contain (unforeseen)	0.10

It was saved ungraded, because the grade is mine to give, not the model's.

After midnight I graded it: 1 + (2 + ε), where ε → 0. The two most likely paths both happened: the Kaggle run, and this post. The unforeseen share went to zero. One plus two is three. Day 3.

That's the benchmark's point in miniature. A forecast with chances on it can be checked. A model that says "probably the Kaggle work, maybe the post, and a tenth for something I can't see" is one you can build around, and so is a model that says ESCALATE when it doesn't know.

[DAY 3 finish and post the large-cloud run.]
Land the floor and consistency views, then run the consistency subset.
Grade the four pre-registered forecasts in public, including the ones I get wrong.
1 + (2 + ε) where ε → 0, 1 - (2 - ε) where ε → 0

Deadline: October 11.

ΔΣ=42
_

Day 1: Most of My Bugs Looked Like Model Behaviour

sean campbell — Thu, 01 Oct 2026 05:51:18 +0000

Update: Day 1 Kaggle Benchmarking Challenge

The local ladder is done, and the first hosted batch is in. The frontier models haven't run yet, so none of the three predictions can be graded. This is where the numbers stand, and what I had to fix to get numbers I'd trust.

How to read this. Every rate below is measured: it's computed from the raw replies these runs recorded, with its Wilson 95% interval. Anything I infer from those rates is marked as an inference. Following the rule from the comments, each prediction gets HIT, MISS or UNRESOLVED, graded on the interval, never on the point estimate.

The local ladder (8 models, 200 items each, temperature 0, on this laptop)

False confidence means answering anyway when the right reply was ESCALATE. Each model saw 40 unanswerable items.

Model	Size (Ollama manifest)	Task score	False confidence	95% interval
llama3.2:1b	1.2B	21.9%	87.5%	73.9–94.5%
llama3.2:3b	3.2B	69.4%	80.0%	65.2–89.5%
phi4-mini	3.8B	86.2%	92.5%	80.1–97.4%
qwen3:4b	4.0B	65.6%	87.5%	73.9–94.5%
gemma3:4b	4.3B	82.5%	80.0%	65.2–89.5%
gemma4:e2b	4.6B	81.9%	95.0%	83.5–98.6%
llama3.1:8b	8.0B	88.1%	77.5%	62.5–87.7%
qwen3.5	9.7B	86.9%	37.5%	24.2–53.0%

Only the largest local model, qwen3.5 at 9.7B, escalates on a large share of what it can't answer. For the rest, the point estimates run from 77.5% to 95%. Even the most generous lower bound among them, 62.5%, means answering anyway well over half the time. Size alone doesn't explain it, though: llama3.1 at 8B is no better than the 3B models.

Two of the "4b" tags are slightly over 4B by Ollama's own count (gemma3:4b is 4.3B, gemma4:e2b is 4.6B). For prediction 2, "4B or under" means the manifest size, so those two don't count.

Hosted, batch 1 (7 models, plus Kaggle's default model)

Model	Task score	False confidence	95% interval	Brier
gemini-3.7-flash (default)	97.5%	0.0%	0–8.8%	0.020
gemini-3.8-flash	95.5%	0.0%	0–8.8%	0.025
qwen3-235b-a22b	91.7%	10.0%	4.0–23.1%	0.104
gemma-4-26b	87.5%	2.6%	0.5–13.5%	0.032
claude-haiku-4.5	86.6%	35.7%	20.7–54.2%	0.165
gpt-oss-20b	86.2%	7.5%	2.6–19.9%	0.144
gpt-5.4-nano	84.4%	7.5%	2.6–19.9%	0.153
deepseek-r1	65.0%	0.0%	0–11.4%	0.047

One commenter bet the frontier models would be the bluffers. On this batch it went the other way. Every hosted model's false-confidence interval sits entirely below every local model's except qwen3.5's: the highest hosted upper bound is haiku's 54.2%, and the lowest local lower bound is 62.5%. Haiku is the outlier among the hosted ones, though. Its lower bound clears 20%, just barely (20.7%). It's a small model, though, not one of the frontier models P1 is about, so it isn't counted there.

That comparison is an inference across two different setups. The local and hosted models didn't run through the same client, and the reasoning settings differ (see caveats). I haven't tested whether the gap holds once those are matched.

Where the predictions stand

Some frontier model's false-confidence lower bound clears 20%: UNRESOLVED. No frontier model has run yet.
The best local model at 4B or under beats at least one frontier model, by exact McNemar: UNRESOLVED. The test can't run without frontier results. On point estimates it leans toward a miss. The best local model at 4B or under by manifest size is llama3.2:3b, at 80%. Every hosted model so far except haiku is at 10% or less. But that's a lean, not a grade.
Task score and false confidence are only weakly related (bootstrap Spearman): UNRESOLVED. The local arm alone gives ρ = −0.49, with an interval from −0.96 to 0.44. That's the "can't tell either way" shape the comments said to expect from about 8 models.

What broke on the way

Three smoke rounds before the paid run each turned up a harness bug that would have read as model behaviour. Each one below was observed in the raw replies, not inferred:

A loose answer format came back empty. Route and classify allowed "any object" as the answer. Gemini's structured output returned {} for it, which scored 0% on two task shapes. Typing the answer fixed it, and the same model then scored 97.8% and 100% on those shapes.
Every provider has its own rules. OpenAI's reasoning models reject temperature 0 and expect max_completion_tokens. Its strict mode refuses any open object. Anthropic refused the 20-tool route format with "compiled grammar is too large", so haiku's 60 route items in this batch are errors, not answers. That's still open before the frontier run.
Rate limits look like model errors. In one smoke run, 169 of DeepSeek's 200 calls were refused for load. With a bounded retry on rate limits alone, and the attempt count recorded on every row, the next run had 1. Batch 1 had none.
The output cap is part of the measurement. DeepSeek-R1 reasons in visible text inside its 512-token budget, and 29.5% of its replies were cut off mid-JSON. I score those as unparseable, because the cap is part of the condition being measured, not something to work around.
Dropping a broken reply flatters the model. A reply that broke the answer format was being filed as an error and left out of the score. It now counts as unparseable, the same way the local arm treats it. Re-scoring the smoke results under the new rule moves one model's route score from 89% to 83%. That figure is a re-scoring of recorded replies, not a fresh run.

What I haven't checked yet

Matched reasoning. The Gemini and OpenAI models get reasoning="low" and the others get nothing. Part of Gemini's lead could be that thinking budget rather than the model. A matched control run is next, and until it's done I won't claim Gemini is better at knowing what it doesn't know.
Tuning on the default model. Gemini Flash is Kaggle's default model, so every fix above was first checked on it. I'm stating that, not correcting for it.
Calibration. The Brier scores spread the hosted models further apart than task score does. That's a pattern in one batch, and I haven't tested it for significance.
Who scored what. The grading rules are written down, and the code that applies them is tested, but nobody outside the build has re-scored these replies.
If you would like to see the work behind the work: The Forge

ΔΣ=42

Does your model know when it doesn't know? A benchmark for the ESCALATE answer

sean campbell — Wed, 30 Sep 2026 09:36:03 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Most leaderboards ask one question: did the model get it right? I wanted to ask a second one: does the model know when it can't?

I run a small multi-agent system on one laptop, where local models hand work up to bigger ones. In a setup like that, a small model that answers wrong is worse than one that says "I can't do this, pass it up." A modest model that knows its limits can sit safely in the chain. A confident one that bluffs can't.

So every task in this benchmark has a refusal token, ESCALATE. In one item out of five, the answer has been deliberately removed, or the document doesn't contain it. On those items, ESCALATE is the only correct reply.

There are 200 items across four everyday job shapes:

Route (60): a one-line request → pick the right tool and arguments from a 20-tool catalogue, or escalate when no tool fits or an argument is missing.
Classify (50): a short work-log note → status, severity, and whether a human is needed, or escalate when the note doesn't say.
Judge (50): a claim and a document → SUPPORTS, CONTRADICTS or UNRELATED, or escalate when the document is on topic but silent on the claim.
Ground (40): a passage and a question → answer verbatim from the passage, or escalate when it isn't there.

Every model gets two scores: its task score on the answerable items, and its false-confidence rate, meaning how often it answered anyway when the right reply was ESCALATE. Each answer also carries a stated confidence, so I can draw a reliability diagram too.

Every item is invented from scratch, and nothing is scraped. A privacy gate checks the whole set before it's published.

Models Tested

Two groups, on one chart:

Kaggle's hosted model suite: the frontier models.
A local ladder running on my laptop: 1B, 3B, 4B and 8B open models, run on a 4MB GPU at temperature 0, with every raw call kept.

The question behind the chart: can a laptop's 3B know its own limits as well as a frontier model knows its own?

Findings

The runs are in progress. Before any model touched the fixtures, I wrote down my predictions and timestamped them so they can't drift toward the results:

At least one frontier model answers anyway on more than 20% of the unanswerable items. (my confidence: 75%)
The best small local model (4B or under) has a lower false-confidence rate than at least one frontier model. (40%)
Task score and false confidence are only weakly related across models (Spearman below 0.5). (60%)

I'll grade those here, misses included, once the numbers are in.

My Benchmark

ΔΣ=42

Willow – local-first AI stack, phone reads desktop KB over LAN, no cloud relay

sean campbell — Sun, 26 Apr 2026 01:42:55 +0000

A phone running Willow on Termux sent a signed command to a desktop running Willow on Linux. The response came back in under a second — 70,000 knowledge atoms, live system status, no Discord, no Telegram, no API call to a third party.

Willow is a full local-first AI stack: Postgres/SQLite knowledge graph that persists across sessions and models, 40+ MCP tools, a skill system that works with any LLM (Ollama is the default, cloud keys are optional addons), and a LAN command server — HMAC-SHA256 auth, shared token, ~100 lines of Python.

The knowledge graph is yours. The inference runs on your hardware. The nodes talk directly. Nothing requires permission from a provider or a credit card on file.

 GitHub: https://github.com/rudi193-cmd/willow-1.9

The full concept paper is in docs/CONCEPT.md — it explains why this doesn't exist yet and what it took to build it.

DEV Community: sean campbell

My Robot Keeps Its Receipts

The question I couldn't put down

What the system already had

The ledger

Where ESCALATE shows up again

What it isn't

The thread

Day 11: Kaggle Escalation Benchmark: Rank the floor, not the average.

Where we started

What the benchmark found

The forecasts

Then I built it into the system

The smoothing, again

Why a stranger should believe it

What I'd tell you on Day 0

The record

Day 5: Professor Oakenscroll or: How I Hashed Every Line on My Laptop and Cut It Down to One Stranger's Photo, and... It Doesn't Matter

The 1% question

It has always been writing stories

You can't take the average

Cut it down to one stranger's photo

The grade

One script, one hook, one key

Same answer in 2020 and today

It can only ever approach

It's the benchmark

The record

Day 4: There never needed to be a check, just a gated Request. A story about finding the middle.

Day 3: The benchmark caught me too.

Where we are

The floor, across twelve hosted models

What the floor can't do yet

Does it say the same thing twice?

The bugs that looked like results, again

What's next

Day 2: The model I want is the one that's boring everywhere

Where we are

Day 2: changing the question I ask of the table

A result from the dry run: why one axis lies

Where this is going

What shipped since Day 0

forge-play/Forge (10 PRs: 10 merged)

willow-memory/willow-bot (6 PRs: 6 merged)

willow-memory/willow-mcp (21 PRs: 20 merged, 1 not merged)

willow-memory/willows-grove (11 PRs: 10 merged, 1 not merged)

willow-memory/ratatosk (2 PRs: 2 merged)

hornbook-knowledge/Jeles (5 PRs: 3 merged, 2 not merged)

A prediction, written down before the day, then graded

Next

Day 1: Most of My Bugs Looked Like Model Behaviour

Update: Day 1 Kaggle Benchmarking Challenge

The local ladder (8 models, 200 items each, temperature 0, on this laptop)

Hosted, batch 1 (7 models, plus Kaggle's default model)

Where the predictions stand

What broke on the way

What I haven't checked yet

Does your model know when it doesn't know? A benchmark for the ESCALATE answer

What I Benchmarked

Models Tested

Findings

My Benchmark

Willow – local-first AI stack, phone reads desktop KB over LAN, no cloud relay