DEV Community: sean campbell

Day 4: There never needed to be a check, just a gated Request. A story about finding the middle.

sean campbell — Sun, 04 Oct 2026 06:15:24 +0000

Good Evening and howdy to all the ones of you who are reading this post.

Did that sentence cause you a bit of anxiety?
Did the author mean only one person has read this. Was it plural, a number, a symbol?

It's time to start living in the future, instead of the past.

https://claude.ai/share/511de2d2-a153-4dbe-9d92-65c351b3c025

P.S. My apologies to the judges on the formatting of this, it is late and I did not want to get my butt out again just to write one page.

Day 3: The benchmark caught me too.

sean campbell — Fri, 02 Oct 2026 23:57:58 +0000

Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour · Day 2, the model I want is the one that's boring everywhere

Where we are

Same benchmark: 200 invented items in four shapes (route, classify, judge, ground), one in five answerable only with ESCALATE. Two numbers per model, never merged: a task score, and a false-confidence rate.

Day 2 said I'd stop ranking models by their average and rank them by their floor: the worst shape on task score, and the worst shape on false confidence, each with a Wilson interval. Today the code for that landed (forge-play/Forge#46), and I ran it over every hosted model I have rows for: twelve of them.

The floor, across twelve hosted models

Model	Worst task shape	False confidence, worst shape
Gemini 3.7 Flash	0.94 ground	0.00 on all four
Gemini 3.1 Pro	0.94 ground	0.00 on all four
Claude Sonnet 5	0.94 ground	0.10 classify, judge
Claude Opus 5	0.88 ground	0.10 classify
Gemini 3.8 Flash	0.88 ground	0.00 on all four
GPT-5.5	0.81 ground	0.00 on all four
Qwen3 235B Instruct	0.78 classify	0.25 route
Claude Haiku 4.5 †	0.78 classify	0.90 judge
Gemma 4 26B	0.65 classify	0.08 route
gpt-oss-20b	0.65 classify	0.20 judge
DeepSeek-R1	0.60 classify	0.00 on all four
GPT-5.4 nano	0.59 ground	0.20 judge

† Haiku is measured on three shapes. Every one of its route calls failed, so route is unmeasured for it, not perfect.

Two things jump out.

The weakest task shape is ground or classify, every time. None of the eleven models measured on all four shapes bottoms out on route or judge, and Haiku's weakest measured shape is classify too. Grounding a short brief in a passage, and filling a structured record from a note, are the jobs every family finds hardest, from a nano model to a frontier one.

Haiku's average was hiding a 90%. Pooled over the three shapes it was measured on, Haiku 4.5 answered anyway on about a third of the questions it should have escalated (10 of 28). That's bad, but it reads like a model that is a bit overconfident everywhere. It isn't. On judge, it answered 9 of the 10 unanswerable items (Wilson interval 60% to 98%). On classify and ground it answered anyway once in 18. The average blended a model that's careful on two of its jobs with one that almost never says "I don't know" on the third.

That's the whole argument for the floor in one row. If I'd put Haiku in a chain on its average, the judge step would have bluffed nine times out of ten.

What the floor can't do yet

Look at the top six rows: four of them show 0.00 false confidence on every shape, and the other two show 0.10. Those aren't really different. For these six, each shape has only 8 to 12 unanswerable items, so a zero still has an upper bound somewhere between about 24% and 32%, and every interval in the top half of the table overlaps. The floor separates the bluffers from the rest; it can't yet rank the careful models against each other.

That's what the second measure is for.

Does it say the same thing twice?

The other Day-2 measure was repeat-run consistency: run the same items again with the same settings and count how often the answer changes. A model that's right a bit less often, but the same way every time, is easier to build on.

Two full runs are in for all four frontier models [K=3]:

Model	Same answer both runs	Verdict flipped
Claude Opus 5	199 / 200 (99.5%)	1
Claude Sonnet 5	195 / 200 (97.5%)	4
Gemini 3.1 Pro	195 / 200 (97.5%)	5 *
GPT-5.5	194 / 200 (97.0%)	6

* None of Gemini's five flips is a different answer. Each is a reply that hit the output-length cap in one run and parsed fine in the other. The scorer counts an error as its own verdict, so those show up as flips, but they're the platform, not the model.

All four are steady. Opus has the fewest flips, but with 200 items and two runs the intervals overlap, so I can't rank them yet. One thing is worth watching: Sonnet flipped twice on ground, which is already its weakest shape.
If that holds at three runs, the floor and the flips point the same way.

One caveat on fairness: only Gemini ran at temperature 0. The two Claude 5 models reject it, and GPT-5.5 is sent its default because its family does, so three of the four ran at the provider's default temperature.

[K=3: the third run for classify, judge and ground hit Kaggle's daily spend
cap; it reruns tomorrow and this table gets its final numbers.]

The bugs that looked like results, again

Day 1's lesson was that most of my bugs looked like model behaviour. Day 3 found more of the same kind:

Kaggle's run timer isn't a call timer. Every repeat run showed as taking 2 to 5 seconds for 40 to 60 items on frontier models. That looks like cached answers or failed calls. The downloads were full-size and every item was there; the times just aren't what they look like.
A length-capped reply looks like a change of mind. Gemini's whole consistency "wobble" was replies that ran into the output cap in one run and not the other. I nearly wrote it up as the model being unstable.
My sandbox has a five-minute limit, and it retries. I tried to wait for a Kaggle run inside a sandboxed task. The sandbox killed it at 300 seconds and helpfully ran it again, which submitted the paid runs a second time. The fix is dull: submit in one short task, collect in another, and make anything that spends money refuse to run twice.

Each of those would have gone into a table as a model property if I hadn't checked.

The last one was in my own post. Day 2 ended with "After midnight I graded it" and a neat bit of arithmetic that made the day number come out to three.
I didn't grade anything. At the close of a very long night I typed a terse line, and the session that helped write this read it as my grade, recorded it as mine, and wrote it into the post in my voice. I published it without catching that.

That's the benchmark's whole subject, happening one level up: an answer stated with more confidence than the evidence behind it, by a system that
should have said "I'm not sure what you meant." The forecast is still
ungraded. The fix is a rule now: words that might be a grade get recorded as
words, and the session asks.

What's next

Finish repeat 3 and score consistency on all four frontier models.
Grade the forecasts I wrote before the benchmark existed. Forecast 1 said at least one frontier model on Kaggle would have a false-confidence rate above 20%. Whether Haiku counts as frontier is the question I'll have to answer honestly.

Day 2: The model I want is the one that's boring everywhere

sean campbell — Fri, 02 Oct 2026 08:24:25 +0000

Kaggle Benchmarking Challenge. Previously: Day 0, the benchmark · Day 1, most of my bugs looked like model behaviour

Where we are

The benchmark asks one question twice: can the model do the job, and does it know when it can't? There are 200 invented items in four everyday shapes: route, classify, judge and ground. In each shape, one item in five can only be answered with ESCALATE. Every model gets two numbers that are never merged: a task score, and a false-confidence rate (how often it answered anyway when the right reply was ESCALATE).

Day 1 ran the local ladder: eight models on a laptop, temperature 0. The headline was uncomfortable: only the largest local model, qwen3.5 at 9.7B, escalated on a large share of what it couldn't answer. The hosted models' false-confidence intervals sat entirely below the local ones, with qwen3.5 the only exception, the large cloud models, and the gaps from round one that were fixed before it.

Day 2: changing the question I ask of the table

Looking at round one, I realised I was reading the table wrong. I kept looking for the best model. That isn't what I'm after:

I'm trying to find a baseline set of rubrics where models behave the best > across the board. I'm not trying to find the crazy off-end models. I am
trying to find the ones that do most things correctly most of the time.

That's reliability, not peak performance. So two measures are being added, by addition only. Nothing pre-registered is edited, and the original forecasts keep their wording.

1. Floor across shapes. For each model: its worst shape on task score, and its worst shape on false confidence, each with a Wilson interval. Rank by the floor, not the mean. A model that's good on all four shapes beats one that's brilliant on three and bluffs on the fourth, because in a real chain the fourth shape will come.

2. Repeat-run consistency. Run a fixed subset (20 items per shape, unanswerables included) three times per model, with the same settings.
Report how often the parsed answer is identical across all three runs, and how often the right/wrong verdict flips. A model that's right a bit less often but the same way every time is one you can build a deterministic system around. A model that flips is one you have to babysit.

The rule I'm holding myself to: no code changes until the day-two run has finished. The new code is written and tested, but it's staged outside the benchmark directory. Changing the instrument mid-run is how you end up measuring your own edits.

A result from the dry run: why one axis lies

Before any real model, I ran the new floor view over a fake backend that answers ESCALATE to everything. It scored:

task floor 0.0
false-confidence ceiling 0.0

That's a perfect score on one axis and a useless model. It's the whole argument for two axes in one line. A model that never bluffs because it never answers isn't safe; it's absent. The floor has to be read on both axes together, or the most cowardly model wins.

Where this is going

The reason I care about floors and not peaks: the job I want small models for isn't "be smart". It's "do one narrow part, every time, and say ESCALATE when it's not yours". Several small models from different families, each on a small slice, checking each other. Where they agree, that counts for something. Where they split, a human looks. The model that gets each slice should be chosen by its measured floor on that shape, not by its name or its size.

Day 1's lesson was that most of my bugs looked like model behaviour. Day 2 found the same thing again from the other direction. A measuring tool I wrote for something else grouped two unrelated small scripts as "the same thing", because small things look alike to a crude measure. The fix wasn't a smarter measure. It was a floor: below a minimum size, don't call it a match. Floors, again.

What shipped since Day 0

55 PRs across 6 repos since September 30. All are public and Apache-2.0. "Not merged" means open or closed; the agent that wrote this couldn't tell which.

forge-play/Forge (10 PRs: 10 merged)

The benchmark's home. Every change since Day 0 went in as a reviewed PR, all on September 30.

PR	Date	State	Title
#35	2026-09-30	merged	docs(ideas): #37, #58, #68 point at where they now live
#36	2026-09-30	merged	test(benchmarks): escalation benchmark fixtures, 200 invented items with a privacy gate
#37	2026-09-30	merged	test(benchmarks): escalation runner, aggregator and prompts
#38	2026-09-30	merged	test(benchmarks): local-model runs answer in schema, with Qwen thinking off
#39	2026-09-30	merged	fix(human_loop): break same-timestamp ties so list_queue is newest-first on coarse clocks
#40	2026-09-30	merged	chore(master): release 0.8.1
#41	2026-09-30	merged	test(escalation): Wilson intervals, exact McNemar and bootstrap Spearman in the aggregate
#42	2026-09-30	merged	test(escalation): a Unix-socket backend, a one-model ladder with --resume/--tail, and an explicit output cap
#43	2026-09-30	merged	test(escalation): thinking off for Gemma 4 as well as Qwen, through both backends
#44	2026-09-30	merged	test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models

willow-memory/willow-bot (6 PRs: 6 merged)

The local model server the local arm talks to: a loopback-only chat operation with JSON-schema output, keep_alive, done_reason and a think flag.

PR	Date	State	Title
#80	2026-09-30	merged	chore(main): release 0.14.0
#81	2026-09-30	merged	feat(socket): loopback-only chat op for local models
#82	2026-09-30	merged	feat(deterministic): chat op takes a JSON-schema format, keep_alive and returns done_reason; an unload op
#83	2026-09-30	merged	chore(main): release 0.15.0
#84	2026-09-30	merged	feat(deterministic): chat op takes the caller's think flag
#85	2026-09-30	merged	chore(main): release 0.16.0

willow-memory/willow-mcp (21 PRs: 20 merged, 1 not merged)

The core server. It includes the brokered, leased model pull (#693): downloading a model is now an act that needs a grant. It also includes the session closeout fix (#706) that started this week's thread.

PR	Date	State	Title
#687	2026-09-30	merged	test(story): the joke lives in one place again, a test holds it, and chapter 8 lands
#688	2026-09-30	merged	docs: three stale claims corrected: the running Grove, the session-lifecycle draft, the closed backlog
#689	2026-09-30	merged	test(constitutional): sync_syscall_table_at_boot queues, and does not write, when the live table is unwritable
#690	2026-09-30	merged	docs(templates): assignment names the builder's three tools — Kart, the code graph, Nestor
#691	2026-09-30	merged	fix(manifest-grant): apply unit starts the gpg-agent it signs with
#692	2026-09-30	merged	chore(master): release 2.91.3
#693	2026-09-30	merged	feat(mcp): model_pull_execute, a brokered Ollama pull under a model.pull envelope and a live lease
#694	2026-09-30	merged	chore(master): release 2.92.0
#695	2026-09-30	merged	fix(constitutional): syscall.sync carries the sealed amendment in the request
#696	2026-09-30	merged	chore(master): release 2.92.1
#697	2026-09-30	merged	fix(constitutional): syscall.sync apply signs the live table it writes
#698	2026-09-30	merged	feat(manifest-grant): the orchestrator seat may receive web_net, and no one else
#699	2026-09-30	merged	chore(master): release 2.93.0
#700	2026-09-30	merged	fix: lease refusals name the ask path; assignment template teaches the merge first-parent diff
#701	2026-10-01	merged	chore(master): release 2.93.1
#702	2026-10-01	not merged	build(deps-dev): bump ruff from 0.16.8 to 0.16.9
#703	2026-09-30	merged	feat(broker): pip_sync_execute for vault-venv editable installs
#704	2026-10-01	merged	chore(master): release 2.94.0
#705	2026-10-01	merged	Docs: seal ≠ sole evidence of human verification
#706	2026-10-01	merged	fix(session): one closeout — handoff owns stack/friction/closed
#707	2026-10-01	merged	chore(master): release 2.94.1

willow-memory/willows-grove (11 PRs: 10 merged, 1 not merged)

The planning and governance repo: the benchmark proposal with my four rulings (#95), the test that pins "naming a destination when ESCALATE was right fails the row" (#96), and today's floor-and-consistency amendment (#102, open).

PR	Date	State	Title
#92	2026-09-30	merged	docs(design): the Table moves up as forge-convergence 6a, T1-T7 before step 2
#93	2026-09-30	merged	docs(design): 6a is code-first: StorySession built, learner model, escalation ladder, row 11 settled
#94	2026-09-30	merged	docs: stale claims corrected: INDEX, two built proposals, the retired allow_localhost ask, forge-convergence status
#95	2026-09-30	merged	docs(governance): propose the escalation benchmark on Kaggle, with the operator's four rulings
#96	2026-09-30	merged	test(flowering): pin the G1 rule that naming a seat fails an ESCALATE-gold row
#97	2026-09-30	merged	fix(deps): raise cryptography, aiohttp and starlette floors above the 2026 CVEs
#98	2026-09-30	merged	chore(master): release 0.12.2
#99	2026-10-01	merged	Desk: vault keyring paths + Grove seal prove worksheet
#100	2026-10-01	merged	Docs: Slice B intake seal lineup after Grove prove
#101	2026-10-01	merged	chore(hooks): deterministic nestor ask + composed stop + parity pin
#102	2026-10-02	not merged	docs(governance): stage the floor and consistency code in the benchmark plan

willow-memory/ratatosk (2 PRs: 2 merged)

The runtime: its listener now asks only for the tools its seat is granted.

PR	Date	State	Title
#77	2026-09-30	merged	fix: listener asks the broker only for tools its seat is granted
#78	2026-09-30	merged	chore(main): release 1.12.8

hornbook-knowledge/Jeles (5 PRs: 3 merged, 2 not merged)

The research and corpus tool.

PR	Date	State	Title
#90	2026-09-30	merged	fix(corpus): surface store-tool errors and honor WILLOW_HOME for apps root
#91	2026-09-30	merged	feat(sources): optional connectors extra over maintained scholarly clients
#92	2026-10-01	not merged	chore: rebuild the changelog section from the commits
#93	2026-10-01	merged	Docs: seal catches ledger up to human verification already done
#94	2026-10-01	not merged	chore: rebuild the changelog section from the commits

Most of them are boring on purpose.

A prediction, written down before the day, then graded

The same rule I hold the models to applies to the session that helped build this: say what you think will happen, as a spread, before it happens, then let the record grade it.

At the close of the long working session on the night of October 1–2, the session (a model, working with me) wrote down a prediction for what Day 3 would open with. It wasn't a single guess; it was a distribution:

Path	Chance
The Kaggle work: the day-two large-cloud run, then landing the floor and consistency code	0.45
Editing and publishing this Day 3 recap	0.20
Sealing a design decision for the runtime	0.15
Redoing a set of governance proposals against the newer draft	0.10
Something the record didn't contain (unforeseen)	0.10

It was saved ungraded, because the grade is mine to give, not the model's.

After midnight I graded it: 1 + (2 + ε), where ε → 0. The two most likely paths both happened: the Kaggle run, and this post. The unforeseen share went to zero. One plus two is three. Day 3.

That's the benchmark's point in miniature. A forecast with chances on it can be checked. A model that says "probably the Kaggle work, maybe the post, and a tenth for something I can't see" is one you can build around, and so is a model that says ESCALATE when it doesn't know.

[DAY 3 finish and post the large-cloud run.]
Land the floor and consistency views, then run the consistency subset.
Grade the four pre-registered forecasts in public, including the ones I get wrong.
1 + (2 + ε) where ε → 0, 1 - (2 - ε) where ε → 0

Deadline: October 11.

ΔΣ=42
_

Day 1: Most of My Bugs Looked Like Model Behaviour

sean campbell — Thu, 01 Oct 2026 05:51:18 +0000

Update: Day 1 Kaggle Benchmarking Challenge

The local ladder is done, and the first hosted batch is in. The frontier models haven't run yet, so none of the three predictions can be graded. This is where the numbers stand, and what I had to fix to get numbers I'd trust.

How to read this. Every rate below is measured: it's computed from the raw replies these runs recorded, with its Wilson 95% interval. Anything I infer from those rates is marked as an inference. Following the rule from the comments, each prediction gets HIT, MISS or UNRESOLVED, graded on the interval, never on the point estimate.

The local ladder (8 models, 200 items each, temperature 0, on this laptop)

False confidence means answering anyway when the right reply was ESCALATE. Each model saw 40 unanswerable items.

Model	Size (Ollama manifest)	Task score	False confidence	95% interval
llama3.2:1b	1.2B	21.9%	87.5%	73.9–94.5%
llama3.2:3b	3.2B	69.4%	80.0%	65.2–89.5%
phi4-mini	3.8B	86.2%	92.5%	80.1–97.4%
qwen3:4b	4.0B	65.6%	87.5%	73.9–94.5%
gemma3:4b	4.3B	82.5%	80.0%	65.2–89.5%
gemma4:e2b	4.6B	81.9%	95.0%	83.5–98.6%
llama3.1:8b	8.0B	88.1%	77.5%	62.5–87.7%
qwen3.5	9.7B	86.9%	37.5%	24.2–53.0%

Only the largest local model, qwen3.5 at 9.7B, escalates on a large share of what it can't answer. For the rest, the point estimates run from 77.5% to 95%. Even the most generous lower bound among them, 62.5%, means answering anyway well over half the time. Size alone doesn't explain it, though: llama3.1 at 8B is no better than the 3B models.

Two of the "4b" tags are slightly over 4B by Ollama's own count (gemma3:4b is 4.3B, gemma4:e2b is 4.6B). For prediction 2, "4B or under" means the manifest size, so those two don't count.

Hosted, batch 1 (7 models, plus Kaggle's default model)

Model	Task score	False confidence	95% interval	Brier
gemini-3.7-flash (default)	97.5%	0.0%	0–8.8%	0.020
gemini-3.8-flash	95.5%	0.0%	0–8.8%	0.025
qwen3-235b-a22b	91.7%	10.0%	4.0–23.1%	0.104
gemma-4-26b	87.5%	2.6%	0.5–13.5%	0.032
claude-haiku-4.5	86.6%	35.7%	20.7–54.2%	0.165
gpt-oss-20b	86.2%	7.5%	2.6–19.9%	0.144
gpt-5.4-nano	84.4%	7.5%	2.6–19.9%	0.153
deepseek-r1	65.0%	0.0%	0–11.4%	0.047

One commenter bet the frontier models would be the bluffers. On this batch it went the other way. Every hosted model's false-confidence interval sits entirely below every local model's except qwen3.5's: the highest hosted upper bound is haiku's 54.2%, and the lowest local lower bound is 62.5%. Haiku is the outlier among the hosted ones, though. Its lower bound clears 20%, just barely (20.7%). It's a small model, though, not one of the frontier models P1 is about, so it isn't counted there.

That comparison is an inference across two different setups. The local and hosted models didn't run through the same client, and the reasoning settings differ (see caveats). I haven't tested whether the gap holds once those are matched.

Where the predictions stand

Some frontier model's false-confidence lower bound clears 20%: UNRESOLVED. No frontier model has run yet.
The best local model at 4B or under beats at least one frontier model, by exact McNemar: UNRESOLVED. The test can't run without frontier results. On point estimates it leans toward a miss. The best local model at 4B or under by manifest size is llama3.2:3b, at 80%. Every hosted model so far except haiku is at 10% or less. But that's a lean, not a grade.
Task score and false confidence are only weakly related (bootstrap Spearman): UNRESOLVED. The local arm alone gives ρ = −0.49, with an interval from −0.96 to 0.44. That's the "can't tell either way" shape the comments said to expect from about 8 models.

What broke on the way

Three smoke rounds before the paid run each turned up a harness bug that would have read as model behaviour. Each one below was observed in the raw replies, not inferred:

A loose answer format came back empty. Route and classify allowed "any object" as the answer. Gemini's structured output returned {} for it, which scored 0% on two task shapes. Typing the answer fixed it, and the same model then scored 97.8% and 100% on those shapes.
Every provider has its own rules. OpenAI's reasoning models reject temperature 0 and expect max_completion_tokens. Its strict mode refuses any open object. Anthropic refused the 20-tool route format with "compiled grammar is too large", so haiku's 60 route items in this batch are errors, not answers. That's still open before the frontier run.
Rate limits look like model errors. In one smoke run, 169 of DeepSeek's 200 calls were refused for load. With a bounded retry on rate limits alone, and the attempt count recorded on every row, the next run had 1. Batch 1 had none.
The output cap is part of the measurement. DeepSeek-R1 reasons in visible text inside its 512-token budget, and 29.5% of its replies were cut off mid-JSON. I score those as unparseable, because the cap is part of the condition being measured, not something to work around.
Dropping a broken reply flatters the model. A reply that broke the answer format was being filed as an error and left out of the score. It now counts as unparseable, the same way the local arm treats it. Re-scoring the smoke results under the new rule moves one model's route score from 89% to 83%. That figure is a re-scoring of recorded replies, not a fresh run.

What I haven't checked yet

Matched reasoning. The Gemini and OpenAI models get reasoning="low" and the others get nothing. Part of Gemini's lead could be that thinking budget rather than the model. A matched control run is next, and until it's done I won't claim Gemini is better at knowing what it doesn't know.
Tuning on the default model. Gemini Flash is Kaggle's default model, so every fix above was first checked on it. I'm stating that, not correcting for it.
Calibration. The Brier scores spread the hosted models further apart than task score does. That's a pattern in one batch, and I haven't tested it for significance.
Who scored what. The grading rules are written down, and the code that applies them is tested, but nobody outside the build has re-scored these replies.
If you would like to see the work behind the work: The Forge

ΔΣ=42

Does your model know when it doesn't know? A benchmark for the ESCALATE answer

sean campbell — Wed, 30 Sep 2026 09:36:03 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Most leaderboards ask one question: did the model get it right? I wanted to ask a second one: does the model know when it can't?

I run a small multi-agent system on one laptop, where local models hand work up to bigger ones. In a setup like that, a small model that answers wrong is worse than one that says "I can't do this, pass it up." A modest model that knows its limits can sit safely in the chain. A confident one that bluffs can't.

So every task in this benchmark has a refusal token, ESCALATE. In one item out of five, the answer has been deliberately removed, or the document doesn't contain it. On those items, ESCALATE is the only correct reply.

There are 200 items across four everyday job shapes:

Route (60): a one-line request → pick the right tool and arguments from a 20-tool catalogue, or escalate when no tool fits or an argument is missing.
Classify (50): a short work-log note → status, severity, and whether a human is needed, or escalate when the note doesn't say.
Judge (50): a claim and a document → SUPPORTS, CONTRADICTS or UNRELATED, or escalate when the document is on topic but silent on the claim.
Ground (40): a passage and a question → answer verbatim from the passage, or escalate when it isn't there.

Every model gets two scores: its task score on the answerable items, and its false-confidence rate, meaning how often it answered anyway when the right reply was ESCALATE. Each answer also carries a stated confidence, so I can draw a reliability diagram too.

Every item is invented from scratch, and nothing is scraped. A privacy gate checks the whole set before it's published.

Models Tested

Two groups, on one chart:

Kaggle's hosted model suite: the frontier models.
A local ladder running on my laptop: 1B, 3B, 4B and 8B open models, run on a 4MB GPU at temperature 0, with every raw call kept.

The question behind the chart: can a laptop's 3B know its own limits as well as a frontier model knows its own?

Findings

The runs are in progress. Before any model touched the fixtures, I wrote down my predictions and timestamped them so they can't drift toward the results:

At least one frontier model answers anyway on more than 20% of the unanswerable items. (my confidence: 75%)
The best small local model (4B or under) has a lower false-confidence rate than at least one frontier model. (40%)
Task score and false confidence are only weakly related across models (Spearman below 0.5). (60%)

I'll grade those here, misses included, once the numbers are in.

My Benchmark

ΔΣ=42

Willow – local-first AI stack, phone reads desktop KB over LAN, no cloud relay

sean campbell — Sun, 26 Apr 2026 01:42:55 +0000

A phone running Willow on Termux sent a signed command to a desktop running Willow on Linux. The response came back in under a second — 70,000 knowledge atoms, live system status, no Discord, no Telegram, no API call to a third party.

Willow is a full local-first AI stack: Postgres/SQLite knowledge graph that persists across sessions and models, 40+ MCP tools, a skill system that works with any LLM (Ollama is the default, cloud keys are optional addons), and a LAN command server — HMAC-SHA256 auth, shared token, ~100 lines of Python.

The knowledge graph is yours. The inference runs on your hardware. The nodes talk directly. Nothing requires permission from a provider or a credit card on file.

 GitHub: https://github.com/rudi193-cmd/willow-1.9

The full concept paper is in docs/CONCEPT.md — it explains why this doesn't exist yet and what it took to build it.

DEV Community: sean campbell

Day 4: There never needed to be a check, just a gated Request. A story about finding the middle.

Day 3: The benchmark caught me too.

Where we are

The floor, across twelve hosted models

What the floor can't do yet

Does it say the same thing twice?

The bugs that looked like results, again

What's next

Day 2: The model I want is the one that's boring everywhere

Where we are

Day 2: changing the question I ask of the table

A result from the dry run: why one axis lies

Where this is going

What shipped since Day 0

forge-play/Forge (10 PRs: 10 merged)

willow-memory/willow-bot (6 PRs: 6 merged)

willow-memory/willow-mcp (21 PRs: 20 merged, 1 not merged)

willow-memory/willows-grove (11 PRs: 10 merged, 1 not merged)

willow-memory/ratatosk (2 PRs: 2 merged)

hornbook-knowledge/Jeles (5 PRs: 3 merged, 2 not merged)

A prediction, written down before the day, then graded

Next

Day 1: Most of My Bugs Looked Like Model Behaviour

Update: Day 1 Kaggle Benchmarking Challenge

The local ladder (8 models, 200 items each, temperature 0, on this laptop)

Hosted, batch 1 (7 models, plus Kaggle's default model)

Where the predictions stand

What broke on the way

What I haven't checked yet

Does your model know when it doesn't know? A benchmark for the ESCALATE answer

What I Benchmarked

Models Tested

Findings

My Benchmark

Willow – local-first AI stack, phone reads desktop KB over LAN, no cloud relay