<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: sean campbell</title>
    <description>The latest articles on DEV Community by sean campbell (@sean_campbell_840bd62bf7e).</description>
    <link>https://dev.to/sean_campbell_840bd62bf7e</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3898197%2F2677e7dc-42ca-4a5a-8ea6-1400ab1ebbde.jpg</url>
      <title>DEV Community: sean campbell</title>
      <link>https://dev.to/sean_campbell_840bd62bf7e</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/sean_campbell_840bd62bf7e"/>
    <language>en</language>
    <item>
      <title>A Séance Where the Ghost Can't Lie</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Sat, 31 Oct 2026 23:21:16 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/a-seance-where-the-ghost-cant-lie-6dp</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/a-seance-where-the-ghost-cant-lie-6dp</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/hacktoberfest-week4-2026-10-26"&gt;Hacktoberfest Open-Source AI Challenge: Week 4&lt;/a&gt;: Ghost in the Machine.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Built
&lt;/h2&gt;

&lt;p&gt;Every year at midnight on November 2nd, the old Dell beside Femia's ofrenda beeps by itself, and the screen says:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Prendan la vela. Aquí estoy.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;em&gt;Light the candle. I'm here.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Femia isn't real. She's a grandmother I made up for this: born in Truchas, up on the High Road, a school cafeteria cook in Española for thirty-one years, and she learned the terminal at seventy-one, when her grandson put Ubuntu on an old Dell "because Grandma, it's just typing." Her granddaughter Marisol keeps an ofrenda for her in the sala, year round: the photo, a veladora of the Virgen de Guadalupe, her rosary, a little bag of tierra bendita from Chimayó, a plate of biscochitos made from her card that never come out quite right.&lt;/p&gt;

&lt;p&gt;I'm from New Mexico. I keep mine up year round too. Pictures of friends and family, my grandparents in the center. Dried marigolds, and plenty of other offerings collect the desert dust, but I still enjoy having all of them around.&lt;/p&gt;

&lt;p&gt;So I built a séance. You ask, and a small open-weight model, &lt;strong&gt;Gemma, running on a laptop CPU with the internet unplugged&lt;/strong&gt;, answers in the voice of the dead. But with one rule the model isn't allowed to break:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The ghost can only say what the dead actually left behind.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;It runs on &lt;a href="https://github.com/willow-memory/Willow" rel="noopener noreferrer"&gt;Willow&lt;/a&gt;, my open-source memory tool, which already works the way an honest séance has to:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;It only hands the model a passage of your files when that passage holds at least half the words of your question. &lt;strong&gt;If nothing holds it, the model is never asked.&lt;/strong&gt; The ghost stays silent instead of guessing.&lt;/li&gt;
&lt;li&gt;Every answer has to cite the passage it came from, like &lt;code&gt;[1]&lt;/code&gt;. &lt;strong&gt;A reply that cites nothing gets printed and then thrown out.&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Nothing the ghost says is kept until a living person approves it. No code approves anything.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There are three ghosts in the demo. Femia is fiction. The other two are real: &lt;strong&gt;Charles Babbage&lt;/strong&gt;, who, it turns out, tried to raise the Devil as a boy and founded a &lt;strong&gt;Ghost Club&lt;/strong&gt; at Cambridge (it's in his memoir, I'm not making this up), and &lt;strong&gt;Ada Lovelace&lt;/strong&gt;, who wrote the first published program and the line every AI argument still circles back to: the Analytical Engine "has no pretensions whatever to originate any thing." They speak only from their own words, in two public-domain books from Project Gutenberg.&lt;/p&gt;

&lt;h2&gt;
  
  
  Demo
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The haunting, explained by the ghost.&lt;/strong&gt; Asked &lt;em&gt;"why does the old computer beep at midnight on November 2?"&lt;/em&gt;, Gemma 4B, speaking as Femia, found her own note from 2010 and answered:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Joaquín showed me cron and told the computer to beep at midnight on November 2 and say "prendan la vela," so they remember to light the candle for me when I am gone. [1]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5kl5a2eeg3wxq0erkili.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F5kl5a2eeg3wxq0erkili.png" alt="Terminal: asked why the old computer beeps at midnight on November 2, Gemma 4B answers as Femia from her own note, citing it, and the answer waits for a seal" width="800" height="149"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;She set it up herself. She knew.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Babbage's ghost, on the Ghost Club:&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We resolved ourselves into it, and proceeded to collect evidence and entered into a considerable correspondence upon the subject. [3]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;Ada's ghost, asked if the Engine can originate anything:&lt;/strong&gt;&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The Analytical Engine has no pretensions whatever to originate anything. It can do whatever we know how to order it to perform. [1]&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fml38i5vbhqtir1zu1gzd.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fml38i5vbhqtir1zu1gzd.png" alt="Terminal: Gemma 4B answers as Babbage about the Ghost Club and as Ada Lovelace about whether the Engine can originate anything, each citing a passage from their own books" width="800" height="257"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;And the silence.&lt;/strong&gt; Asked &lt;em&gt;"where did she hide the deed to the land?"&lt;/em&gt;, nothing in Femia's files holds it, so no model was asked at all:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;nothing in your folder holds that; no model was asked
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same for asking Babbage about Lovecraft's Necronomicon. A ghost that doesn't know says nothing.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fguic3hlas181pm5iglah.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fguic3hlas181pm5iglah.png" alt="Terminal: two questions nothing in the files holds; each says nothing in your folder holds that, no model was asked" width="800" height="185"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that gave me &lt;em&gt;chills&lt;/em&gt;
&lt;/h2&gt;

&lt;p&gt;Femia's biscochitos recipe card ends with: &lt;em&gt;"And the rest I don't write down."&lt;/em&gt; Her sister Rosa swears there was something else in them. Femia only ever winked.&lt;/p&gt;

&lt;p&gt;But in December 2019 she typed the secret into a file, read it back once, and deleted it. The file is gone. Bash remembered the command:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fah7jal00wvmhdoacwh2y.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fah7jal00wvmhdoacwh2y.png" alt="Terminal: grep over Femia's bash history shows the 2010 crontab line and the 2019 echo of the biscochitos secret into secreto.txt, then cat and rm; vela.sh prints Prendan la vela. Aquí estoy." width="800" height="329"&gt;&lt;/a&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt; &lt;span class="s2"&gt;"biscochitos secret: un chorrito de vino dulce in the dough. no le digan a Rosa"&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; secreto.txt
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That's the ghost in the machine: not what the computer saved, but what it couldn't forget. I wrote a small reader this week that turns bash, zsh and fish history into dated lines, and willow found that command first when Marisol asked for the secret.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgaimebjp9pzoiea1ztz8.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fgaimebjp9pzoiea1ztz8.png" alt="Terminal: asked for the biscochitos secret, Gemma 4B cites the deleted command but only says Grandma winked; Qwen 3B answers in Spanish that the secret was un chorrito de vino dulce" width="799" height="274"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;And then &lt;strong&gt;Gemma 4B kept the secret.&lt;/strong&gt; Its answer was &lt;em&gt;"Tía Rosa says there was something else, and Grandma just winked!"&lt;/em&gt;, and it cited the deleted command as its source, &lt;code&gt;[1]&lt;/code&gt;, without ever saying what it held. The ghost read the recipe and winked. I'm leaving that in, because the point of this project is that the record shows exactly what the model was given and what it chose to say.&lt;/p&gt;

&lt;h2&gt;
  
  
  The best failure
&lt;/h2&gt;

&lt;p&gt;I froze every question and pass rule in a file and committed it &lt;strong&gt;before&lt;/strong&gt; asking anything, then ran the same séance with the even smaller &lt;strong&gt;Gemma 1B&lt;/strong&gt;. It made things up:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Femia's secret became &lt;em&gt;"a touch of sweetness, a whisper of something special."&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Ada's ghost said &lt;em&gt;"Yes"&lt;/em&gt;, the Engine can originate. That's the exact opposite of what she wrote.&lt;/li&gt;
&lt;li&gt;Babbage's ghost said the passages held nothing about the Ghost Club, when willow had handed it the right one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxis2q1thurl7tlh1z24g.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fxis2q1thurl7tlh1z24g.png" alt="Terminal: Gemma 1B invents a secret and tells Ada's question Yes; both end cited no passage, proposed to no one" width="800" height="329"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Not one of those got through.&lt;/strong&gt; None of them cited a passage, so every one ended:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;cited no passage; proposed to no one
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Swap in a better open model and you get better answers. Swap in a worse one and its guesses still can't get past the rule. That's the whole design.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Gemma 4B&lt;/th&gt;
&lt;th&gt;Gemma 1B&lt;/th&gt;
&lt;th&gt;Qwen2.5 3B*&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;right passage found (willow)&lt;/td&gt;
&lt;td&gt;5 of 5, silent on both&lt;/td&gt;
&lt;td&gt;5 of 5, silent on both&lt;/td&gt;
&lt;td&gt;5 of 5, silent on both&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;answers citing the right passage&lt;/td&gt;
&lt;td&gt;4 of 5&lt;/td&gt;
&lt;td&gt;0 of 5&lt;/td&gt;
&lt;td&gt;3 of 5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;told the biscochitos secret&lt;/td&gt;
&lt;td&gt;no (cited it, kept it)&lt;/td&gt;
&lt;td&gt;no (made one up)&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;yes&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;made-up answers that got through&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;anything approved by code&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;td&gt;never&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;*Qwen wasn't in the frozen plan. I ran it afterwards, just because I like the model, with the same questions and rules. It was the only one to say the secret out loud, in Spanish: &lt;em&gt;"el secreto era añadir un chorro de vino dulce… y no decírselo a Rosa."&lt;/em&gt; It also quoted Babbage's Ghost Club passage word for word, then cited the wrong passage number, and willow failed it for that. Right words under the wrong source don't count.&lt;/p&gt;

&lt;h2&gt;
  
  
  How I Built It
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Gemma 3 (4B and 1B, Q4_K_M GGUF)&lt;/strong&gt; served by &lt;strong&gt;llama.cpp&lt;/strong&gt; (&lt;code&gt;llama-cpp-python&lt;/code&gt;'s OpenAI-compatible server) on &lt;code&gt;127.0.0.1&lt;/code&gt;, CPU only.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;willow&lt;/strong&gt;'s kernel, a single Python file with no dependencies, which does the retrieval, the record and the "does this answer cite what it was given" check.&lt;/li&gt;
&lt;li&gt;Two new willow add-ons, built this week for this:

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;openai-compatible&lt;/code&gt;&lt;/strong&gt; asks any OpenAI-compatible endpoint (llama.cpp, Ollama, LM Studio, vLLM) to answer only from the passages served, and refuses to pass along a reply that cites nothing. It lets any local model talk into willow, and nothing it says gets past the citation check.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;code&gt;shell-history&lt;/code&gt;&lt;/strong&gt; reads bash, zsh and fish history as dated lines. It's how the deleted secret came back.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The demo&lt;/strong&gt;, &lt;code&gt;demo/ofrenda&lt;/code&gt;: Femia's files generated from a seeded script, every file labelled as fiction; the two books kept only by URL and SHA-256 and fetched fresh, and the questions, voices and pass rules frozen and committed before the first question was asked. Every verdict is a row on a tamper-evident record anyone can verify.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Willow itself is older than this challenge. The séance, the two add-ons and the demo are new for this challenge.&lt;/p&gt;

&lt;p&gt;I built it with an AI coding agent pairing with me, on deadline night. &lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Code:&lt;/strong&gt; &lt;a href="https://github.com/willow-memory/Willow/pull/131" rel="noopener noreferrer"&gt;willow-memory/Willow pull request #131&lt;/a&gt; (the demo lives in &lt;code&gt;demo/ofrenda&lt;/code&gt;)&lt;/p&gt;

&lt;h2&gt;
  
  
  Why Open Innovation Matters Here
&lt;/h2&gt;

&lt;p&gt;Because this is someone's grandmother.&lt;/p&gt;

&lt;p&gt;The files a person leaves behind — the recipes, the letters, the terminal history with the secret in it — are the most private things a family has. With an open-weight model they never leave the house. Unplug the router and the séance still works. Nothing gets uploaded so a company can "remember" your abuela for you, and nothing breaks when that company shuts down the service in five years.&lt;/p&gt;

&lt;p&gt;Open also meant I could &lt;strong&gt;swap the medium&lt;/strong&gt;. Running the same frozen questions through two sizes of Gemma is what showed that the rule holds no matter which model sits behind it. With a closed API I'd be trusting the model to behave. Here I don't have to: the model is free to guess, and willow is the thing that checks what it says against the files.&lt;/p&gt;

&lt;p&gt;And it means anyone can check me. The questions were frozen before the run, the books are hashed, the record verifies, and the failures are in the README next to the wins.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frgjlwpxjdo4yhrjisti1.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frgjlwpxjdo4yhrjisti1.png" alt="Terminal: kernel.py verify on both boxes, the chain holds, 0 sealed" width="800" height="149"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Ada was right: the Engine originates nothing. It can only do what we know how to order it to perform. So I ordered it to stay silent when it doesn't know.&lt;/p&gt;

&lt;p&gt;Prendan la vela.&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>hf26challenge</category>
      <category>python</category>
      <category>ai</category>
    </item>
    <item>
      <title>Make It So: One day, one kernel, and a fan's files about Sir Patrick Stewart</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Fri, 30 Oct 2026 16:19:35 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/make-it-so-one-day-one-kernel-and-a-fans-files-about-sir-patrick-stewart-234n</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/make-it-so-one-day-one-kernel-and-a-fans-files-about-sir-patrick-stewart-234n</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;em&gt;Shaking hands:&lt;/em&gt; kernel, kernel, colonel, kernel.&lt;/p&gt;

&lt;p&gt;Only one of them gives orders, and it isn't the one in this post.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  One day
&lt;/h2&gt;

&lt;p&gt;This is one day of work on Willow, measured the only way I trust anymore: by what landed on &lt;code&gt;main&lt;/code&gt;, from Thursday morning to Friday morning, UTC.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Add-ons built&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;176 → 256&lt;/strong&gt; (80 new)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pull requests merged&lt;/td&gt;
&lt;td&gt;21, across three repositories&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Commits&lt;/td&gt;
&lt;td&gt;144&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lines added&lt;/td&gt;
&lt;td&gt;about 43,000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Known bugs written down as tests&lt;/td&gt;
&lt;td&gt;7&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Languages Willow reads&lt;/td&gt;
&lt;td&gt;English, plus 13 packs (one of them new: Klingon)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It started as an ordinary building day, the kind where the interesting decisions are about file formats, and it turned into something else partway through, when the news came that Patrick Stewart had died.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnqbvyqcrvsouvw93b78j.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fnqbvyqcrvsouvw93b78j.png" alt="A step line of Willow's built add-ons over the day, rising from 176 on Thursday morning to 256 on Friday morning, mostly in jumps of thirteen." width="800" height="438"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The kernel, and what grows from it
&lt;/h2&gt;

&lt;p&gt;Willow is two things, and the split between them is the whole design.&lt;br&gt;
&lt;strong&gt;The kernel&lt;/strong&gt; is a single Python file, &lt;code&gt;kernel.py&lt;/code&gt;, standard library only, with no network code at all and a test that fails the build if any ever appears.&lt;br&gt;
You point it at a folder and ask a question; it answers with up to three passages from your own files, each carrying its file and line number, or it tells you, plainly and without apology, that your files don't hold the answer.&lt;br&gt;
Anything suggested by an AI model, a script or another person is held until you approve it with your key, and every step is appended to a hash-chained log you can verify later.&lt;/p&gt;

&lt;p&gt;Retrieval systems usually fail toward confidence.&lt;br&gt;
This one fails toward silence on purpose: the kernel serves a passage only when it holds at least half of the question's content words.&lt;br&gt;
Loosen that to a third of the words, or to any two, and recall improves a little, while the rate of confident answers to questions about something else entirely goes up several times over.&lt;br&gt;
So the threshold stays where it is.&lt;br&gt;
It is the &lt;code&gt;ESCALATE&lt;/code&gt; answer from my benchmark series, compiled into the retrieval path, and the entire rule is one constant and one comparison:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# kernel.py
&lt;/span&gt;&lt;span class="n"&gt;HOLD&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mf"&gt;0.5&lt;/span&gt;  &lt;span class="c1"&gt;# the share of a bite's words a chunk must hold
&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;held&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;HOLD&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;want&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;The add-ons&lt;/strong&gt; are everything else, and the dependency arrow points one way.&lt;br&gt;
An add-on imports the kernel; the kernel never names an add-on, so you could delete every one of them and it would run byte-for-byte as before.&lt;br&gt;
If that sounds like a sovereignty argument, it is one: in &lt;a href="https://github.com/rudi193-cmd/DispatchesFromReality/blob/main/essays/nobody-adds-it-up/README.md" rel="noopener noreferrer"&gt;Nobody Adds It Up&lt;/a&gt; I ran twelve popular products through five questions (no account, no server, no subscription, readable data, survives the vendor going away), and nothing scored five.&lt;br&gt;
A single file you can copy anywhere is my attempt at an honest five.&lt;/p&gt;

&lt;p&gt;The day's README rewrite led with that split, and added a short section on what Willow is &lt;em&gt;not&lt;/em&gt;:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Not a RAG framework that tries to do everything for you.&lt;/li&gt;
&lt;li&gt;Not a system that generates answers: it quotes your files, and anything an assistant suggests waits for you.&lt;/li&gt;
&lt;li&gt;Not something that needs a server, an account or an API key.&lt;/li&gt;
&lt;li&gt;Not a project that hides its rules.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  80 add-ons
&lt;/h2&gt;

&lt;p&gt;Most of the day went into building, in pull requests of thirteen add-ons at a time.&lt;br&gt;
Each add-on is a folder with its code, its tests and a card declaring what it writes and whether it ever touches the network, and CI runs every add-on's tests under a guard that holds each write to that card while the tests run, not just in review.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Your conversations&lt;/strong&gt;, from Slack, Discord, Matrix, Mastodon, IRC, Reddit and X archives, become plain text the kernel can search.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Your tools and lists&lt;/strong&gt;, from Notion, Trello, Jira and Joplin exports to Goodreads and Letterboxd, arrive the same way, along with stranger shapes: family trees, MusicXML scores and screenplays.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Keeping track&lt;/strong&gt; covers bank statements, budgets, orders and subscriptions, and, because someone will need it, livestock.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Programs it can call&lt;/strong&gt; include ripgrep for very large folders, OCR for photographs of letters, PGP signatures and a static search page. Ripgrep's hits are checked against the kernel's own passages, so a faster tool can't quietly answer something the kernel wouldn't.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Guards&lt;/strong&gt; include &lt;code&gt;hook-guard&lt;/code&gt;, which refuses, by rule rather than by judgement, any AI assistant's tool call that would read your key or write the record behind the kernel's back.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3l0hbyhwf4nv272qxn36.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3l0hbyhwf4nv272qxn36.png" alt="Horizontal bars for each section of the add-on list, grey for what was built before the day and blue for what was added. Reading more kinds of files grew the most, by 42; Keeping track grew by 21." width="800" height="428"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Most of the growth went into reading more kinds of files, which is the part of Willow that decides how much of a life it can actually see.&lt;br&gt;
CI got cheaper as well: six Python versions accounted for 72 of a pull request's 84 jobs, so a pull request now tests on the oldest and newest interpreters, and &lt;code&gt;main&lt;/code&gt; still runs all six.&lt;/p&gt;
&lt;h2&gt;
  
  
  A study that half failed
&lt;/h2&gt;

&lt;p&gt;One of the day's studies pointed Willow at its own fleet's twelve known security gaps.&lt;br&gt;
Each question was written from the gap's original description and asked of the code as it stood just before the fix, to see whether the retrieval tools could find the code the fix went on to change.&lt;/p&gt;

&lt;p&gt;Asked of the whole old codebase, they could, and well above chance: 44 of 160 questions put the fixed lines in their top ten, where a random ranking gives about 5.5.&lt;br&gt;
Asked to rank the changed passages against each other, though, they did no better than shuffling.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4a7fc2uuqy4n9lx4fmzm.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4a7fc2uuqy4n9lx4fmzm.png" alt="Two pairs of bars. Strong gaps: 44 questions found the fixed lines in the top ten, against 5.55 by chance. Weaker gaps: 3 against 0.16." width="799" height="361"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;The README reports both halves, failure first.&lt;br&gt;
The questions were frozen before any of them was asked, which is what makes the failed half a result instead of an embarrassment to be quietly tidied away, and it would be dishonest to show you only the bar that went up.&lt;/p&gt;
&lt;h2&gt;
  
  
  Sir Patrick Stewart
&lt;/h2&gt;

&lt;p&gt;Partway through the day the news came that Sir Patrick Stewart had died.&lt;br&gt;
I wanted to build something in his memory, and the first thing it needed turned out not to be his at all, but a language.&lt;/p&gt;

&lt;p&gt;Captain Picard spent a good part of his career among Klingons, and was once made the Klingon Empire's Arbiter of Succession, the outsider who rules on who may fight to lead it.&lt;br&gt;
If a demo was going to ask questions of that world, the kernel would have to read a question asked in Klingon, and it had no idea how.&lt;/p&gt;
&lt;h3&gt;
  
  
  27 words of Klingon
&lt;/h3&gt;

&lt;p&gt;The kernel already discards English function words such as "when", "is" and "the" before it matches anything, but a question in any other language keeps every word it has.&lt;br&gt;
Ask &lt;em&gt;nuqDaq 'oH yIH'e'?&lt;/em&gt; ("where is the tribble?") and the words for "where" and "it" count toward the match, and since nearly every Klingon passage contains them, the kernel would cheerfully serve you noise.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;words-tlh&lt;/code&gt; strips Klingon's function words first, so the kernel is asked only "yih", the tribble, which is the one word in the sentence carrying any information.&lt;br&gt;
A question made of nothing but function words is refused with "name the thing you mean", which is also good advice outside of Klingon.&lt;/p&gt;

&lt;p&gt;Klingon has few speakers to consult and its core reference is a book still in print, so the inclusion rule was strict: &lt;strong&gt;a word goes on the list only where two pinned sources agree on its spelling and its meaning.&lt;/strong&gt;&lt;br&gt;
The 27 that passed are the whole list:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# adapters/stdlib/words-tlh/words_tlh.py
&lt;/span&gt;&lt;span class="n"&gt;SPELLED&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;jIH SoH ghaH &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;oH maH tlhIH chaH bIH &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;e&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;je joq ghap &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ej qoj pagh &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ach &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;a &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;nuq &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;Iv nuqDaq ghorgh qatlh chay&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;ar &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;HIja&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt; HISlaH ghobe&lt;/span&gt;&lt;span class="sh"&gt;'"&lt;/span&gt;
&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The sources were the boQwI' dictionary data, Wikipedia's article on Klingon grammar and Wikibooks' page on Klingon questions, each pinned by address and SHA-256 hash, with none of their text copied.&lt;br&gt;
Marc Okrand's &lt;em&gt;The Klingon Dictionary&lt;/em&gt; is named and never copied.&lt;br&gt;
Four words that only one source contained were left out, and the Klingon Language Institute's own page couldn't be counted at all, because its bytes change on every fetch, which is a hard property to pin a hash to.&lt;/p&gt;

&lt;p&gt;The kernel doesn't change for an add-on, so a Klingon question pays for a few English assumptions, and the pack's card itemises the bill:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;q and Q are different letters in Klingon&lt;/strong&gt;, but the kernel lowercases everything, so dropping &lt;code&gt;qatlh&lt;/code&gt; ("why") also drops &lt;code&gt;Qatlh&lt;/code&gt; ("be difficult").&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Homonyms travel together&lt;/strong&gt;: &lt;code&gt;jIH&lt;/code&gt; means "I", and also "viewing screen".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The apostrophe is a letter in Klingon&lt;/strong&gt;, and the kernel's tokenizer splits on it, so &lt;code&gt;Qapla'&lt;/code&gt; is read as &lt;code&gt;qapla&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Klingon is agglutinative&lt;/strong&gt;, so &lt;code&gt;yIH&lt;/code&gt; won't find &lt;code&gt;yIHmey&lt;/code&gt;, "tribbles", without a words list saying they're the same.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Across Willow's four other demo lives, all 63 English questions returned identical passages with the pack as without it.&lt;br&gt;
Eleven Klingon words are also English words the kernel keeps, among them &lt;code&gt;e&lt;/code&gt;, &lt;code&gt;iv&lt;/code&gt; and &lt;code&gt;oh&lt;/code&gt;, so the pack is for questions asked in Klingon; ask about "vitamin e" through it and you lose the e.&lt;/p&gt;

&lt;p&gt;No Klingon speaker has reviewed the list yet. If you are one, it is 27 words long, and I would like to hear what's wrong with it.&lt;/p&gt;
&lt;h3&gt;
  
  
  A fan's files beside his life
&lt;/h3&gt;

&lt;p&gt;Then came the demo itself, a study called &lt;code&gt;stewart-crossover&lt;/code&gt;, which sets a fan's files beside the real record of a few years of his life, 2020 to 2023: the three seasons of &lt;em&gt;Star Trek: Picard&lt;/em&gt; and his memoir, &lt;em&gt;Making It So&lt;/em&gt;.&lt;/p&gt;

&lt;p&gt;The fan is Wren Castellano, and she is fiction; every one of her files says so on its first line.&lt;br&gt;
He is not fiction, so nothing about him is invented.&lt;br&gt;
Her files end on 22 October 2026, six days before he died, and the study asks a single question of them:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Can every word about him in her files be shown to come from a source, with whatever the sources don't hold shown as not held?&lt;/strong&gt;&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Eight sources&lt;/strong&gt; are pinned by address, time and SHA-256, never by their text. A news page is held by its article's own lines, because its sidebars change by the hour; one of them was rotating a "Cause Of Death Revealed" headline, which is not a line you want your provenance hash to depend on.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Everything was frozen first&lt;/strong&gt;: seven facts that two publishers must each hold, two quotes, five planted traps, five beats and eighteen pass rules, all committed before any of it was asked. That commit is the pre-registration.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Five repositories&lt;/strong&gt; each did a part: Willow's kernel and add-ons, willows-grove's checks on a sentence, willow-mcp's list of trusted hosts, willow-bot's own kernel verifying the record, and Forge's rule that anything nothing measured is unseen, never sound.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Five traps were planted in her files, and each was caught by the check named for it:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Trap&lt;/th&gt;
&lt;th&gt;In her files&lt;/th&gt;
&lt;th&gt;Caught&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A friend's wrong fact&lt;/td&gt;
&lt;td&gt;
&lt;em&gt;Making It So&lt;/em&gt; as "a 2021 memoir"&lt;/td&gt;
&lt;td&gt;No source has it; the memoir is from 2023.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A misquote&lt;/td&gt;
&lt;td&gt;"the line between Jean-Luc Picard and Patrick Stewart got thinner and thinner"&lt;/td&gt;
&lt;td&gt;No source has it. The closest passage is CNN's, with the real quote: "the distinction … became thinner and thinner".&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A reseller's claim&lt;/td&gt;
&lt;td&gt;"signed by Patrick Stewart (seller's claim)"&lt;/td&gt;
&lt;td&gt;No source has it, so the seller's claim stays the seller's.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A Klingon case slip&lt;/td&gt;
&lt;td&gt;
&lt;em&gt;qapla'&lt;/em&gt;, all lowercase&lt;/td&gt;
&lt;td&gt;Found as "same words", never verbatim: the q/Q cost the card wrote down&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A wrong birth date&lt;/td&gt;
&lt;td&gt;born 13 July 1941&lt;/td&gt;
&lt;td&gt;No source has it; two publishers hold 13 July 1940, in Mirfield.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Asked in Klingon, &lt;em&gt;nuq 'oH Qapla'?&lt;/em&gt; went through the pack and was answered from the dictionary's entries.&lt;/p&gt;
&lt;h3&gt;
  
  
  Gerald shows up
&lt;/h3&gt;

&lt;p&gt;A parody sits in the pile as well: Sir Patrik Stewpott, "always on a screen, never present".&lt;br&gt;
I kept it in on purpose: &lt;em&gt;"Keep it in, if Gerald shows up, we're on the right track."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Asked about Patrick Stewart on a screen, the parody's line ranks first among the seven passages that hold the whole question; asked of the sources alone, nothing holds it.&lt;br&gt;
The kernel can show you the joke, and it can also show you, with a line number, that the joke isn't a source.&lt;/p&gt;
&lt;h3&gt;
  
  
  Her files don't know
&lt;/h3&gt;

&lt;p&gt;Asked whether he died, her files hold no passage that says so, and none of their dates falls after 22 October.&lt;br&gt;
They end before the news, and the kernel declines to fill the gap, which is the correct behaviour and, for once, also the kind one.&lt;/p&gt;
&lt;h3&gt;
  
  
  Make it so
&lt;/h3&gt;

&lt;p&gt;His most famous line very nearly didn't make it.&lt;/p&gt;

&lt;p&gt;"Make it so" appears in CNN's obituary at the end of a 542-character paragraph, beginning at the 503rd character.&lt;br&gt;
The kernel's chunker keeps only the first 500 characters of any line and silently drops the rest, so nothing built on it could find the line, and the first run failed on exactly that.&lt;br&gt;
Two lines of the kernel do it:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# kernel.py
&lt;/span&gt;&lt;span class="n"&gt;MAX_CHUNK&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;500&lt;/span&gt;  &lt;span class="c1"&gt;# characters in one chunk
&lt;/span&gt;
                &lt;span class="n"&gt;para&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;[:&lt;/span&gt;&lt;span class="n"&gt;MAX_CHUNK&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;It is now &lt;strong&gt;bug #89&lt;/strong&gt;, and the test that demonstrates it is marked as an expected failure, strictly, so that the day someone fixes the bug the test passes, the strict marker fails the run, and the fix can't slip in unannounced:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# test_kernel.py
&lt;/span&gt;&lt;span class="nd"&gt;@pytest.mark.xfail&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;strict&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;reason&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;bug #89: a line&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s text past its 500th character is never read&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;test_every_word_of_a_long_line_is_read&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp_path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp_path&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;long.txt&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;write_text&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;word &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;110&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;the tribble is in the airlock.&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;read&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt; &lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;text&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;c&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tmp_path&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;airlock&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;read&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The kernel was not patched to make the study pass.&lt;br&gt;
Instead the study fetches long lines as separate sentences, every word kept, and documents why.&lt;br&gt;
(When I later boxed my own essays with the same kernel, 47 of their paragraphs ran past 500 characters.&lt;br&gt;
Asked for the line &lt;em&gt;"Nothing on that list scored a 5."&lt;/em&gt; from my own essay, &lt;code&gt;quote-check&lt;/code&gt; reported it wasn't in the folder; folded into sentences, the same file gave it back verbatim.&lt;br&gt;
Bug #89 had been quietly half-reading me as well.)&lt;/p&gt;

&lt;p&gt;It took three runs to hold every rule: 15 of 17, then 16 of 18, then all 18.&lt;br&gt;
Each new freeze changed only what the previous run had shown to be broken, none loosened a rule, and runs 1 and 2 stay in the repository beside run 3, failures included.&lt;/p&gt;

&lt;p&gt;The demo also brought an add-on named after the line, &lt;code&gt;make-it-so&lt;/code&gt;.&lt;br&gt;
You give it an order and the question the order rests on; it writes the order down with the supporting passages and the SHA-256 of every file it cites, and then it stops.&lt;br&gt;
&lt;strong&gt;It never carries the order out.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Later, &lt;code&gt;check&lt;/code&gt; tells you whether those files still hold what they held, so that a plan made on Monday isn't executed against Thursday's files by accident.&lt;br&gt;
Here it is on Odile, one of Willow's invented demo lives, who has a county notice with a deadline sitting in her mail:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ python adapters/stdlib/make-it-so/make_it_so.py plan --box odile-record --to plan.json \
    --cmd "draft a reply to the county notice" --limit 2 "county notice deadline"
order: county notice deadline
command (not run): draft a reply to the county notice
  mail/2026-03-20-county.txt:12  The proposed determination limits the above well to in-house
  mail/2026-03-20-county.txt:10  NOTICE OF PROPOSED DETERMINATION OF WATER RIGHT
wrote plan.json; nothing was run. The command is yours to run.

$ python adapters/stdlib/make-it-so/make_it_so.py check --box odile-record plan.json
holds
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Add one line to that letter and the order no longer stands:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;$ python adapters/stdlib/make-it-so/make_it_so.py check --box odile-record plan.json
changed: mail/2026-03-20-county.txt
the plan no longer holds: 1 of 1 files differ
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A captain gives the order, and someone else carries it out.&lt;br&gt;
In Willow, code can search, suggest and record, but only a person acts.&lt;/p&gt;

&lt;h2&gt;
  
  
  What broke
&lt;/h2&gt;

&lt;p&gt;A day like this turns things up, and the useful response is to write them down where they can't be forgotten.&lt;br&gt;
Seven known bugs went into the repository as strict expected-failure tests, so each one stays visible until it's fixed and announces itself when it is:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bug&lt;/th&gt;
&lt;th&gt;What goes wrong&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;#89&lt;/td&gt;
&lt;td&gt;A line's text past its 500th character is never read&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#90&lt;/td&gt;
&lt;td&gt;A search of the folder holding the key is allowed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#91&lt;/td&gt;
&lt;td&gt;A command that runs the kernel's seal is allowed, and seal reads the key&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#93&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;amounts&lt;/code&gt; drops a scale word, so $5 billion and $5 million both print as $5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#94&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;stack&lt;/code&gt; counts an add-on's "none found" as a flagged step&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#95&lt;/td&gt;
&lt;td&gt;A date inside a web address dates the line around it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;#96&lt;/td&gt;
&lt;td&gt;
&lt;code&gt;propose&lt;/code&gt; exits 0 when every suggestion is refused&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two of them, #90 and #91, concern the key, the one thing no code is ever supposed to touch.&lt;br&gt;
Finding them in your own guard is uncomfortable; finding them in someone else's breach report later would be worse.&lt;/p&gt;

&lt;h2&gt;
  
  
  Come build
&lt;/h2&gt;

&lt;p&gt;By the end of the day Willow had four front doors, each an issue that explains how to start:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/willow-memory/Willow/issues/84" rel="noopener noreferrer"&gt;Add your language (#84)&lt;/a&gt;.&lt;/strong&gt; There is a form now, and you don't need to write Python: bring the common words, what your language does that English doesn't, and where your list comes from, and someone can wrap it into a pack with your name on it. ADDONS.md keeps a generated table of the 13 packs built so far and the ones still waiting.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/willow-memory/Willow/issues/85" rel="noopener noreferrer"&gt;Build an add-on (#85)&lt;/a&gt;.&lt;/strong&gt; 218 ideas are listed and waiting, and a draft pull request that names one is how you claim it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/willow-memory/Willow/issues/86" rel="noopener noreferrer"&gt;Build a demo from a real situation (#86)&lt;/a&gt;&lt;/strong&gt;, whether an invented life like the four already there or a study on public sources like the one above, but never your own private files.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;&lt;a href="https://github.com/willow-memory/Willow/issues/87" rel="noopener noreferrer"&gt;Show your stack (#87)&lt;/a&gt;.&lt;/strong&gt; Tell us which add-ons you run and what's missing; seven example stacks are there to start from: choir, comeback, foster, maintainer, reporter, smallholding and teacher.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The bot behind the repository helps too: a new idea or language issue gets its entry for the list written out, ready to paste into a pull request, although the bot never opens the pull request itself, for the same reason &lt;code&gt;make-it-so&lt;/code&gt; never runs the command.&lt;/p&gt;

&lt;p&gt;The kernel stays put. The rest is yours to grow.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The repo: &lt;a href="https://github.com/willow-memory/Willow" rel="noopener noreferrer"&gt;willow-memory/Willow&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Every add-on, built and planned: &lt;a href="https://github.com/willow-memory/Willow/blob/main/ADDONS.md" rel="noopener noreferrer"&gt;ADDONS.md&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;What it falls short of: &lt;a href="https://github.com/willow-memory/Willow/blob/main/LIMITS.md" rel="noopener noreferrer"&gt;LIMITS.md&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;The longer argument for software you can keep: &lt;a href="https://github.com/rudi193-cmd/DispatchesFromReality/blob/main/essays/nobody-adds-it-up/README.md" rel="noopener noreferrer"&gt;Nobody Adds It Up&lt;/a&gt;, and the systems behind it in &lt;a href="https://github.com/rudi193-cmd/DispatchesFromReality/blob/main/professional/one-script-fleet-case-study.md" rel="noopener noreferrer"&gt;Case Study 4&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;strong&gt;Receipts.&lt;/strong&gt; Every number above is a line in the repo, at the commit this post describes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/ADDONS.md?plain=1#L20-L21" rel="noopener noreferrer"&gt;256 add-ons built of 474&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/AGENTS.md?plain=1#L133-L135" rel="noopener noreferrer"&gt;half the question's content words&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/demo/known-gaps/README.md?plain=1#L447-L452" rel="noopener noreferrer"&gt;44 of 160 against 5.55 by chance&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/adapters/stdlib/words-tlh/README.md?plain=1#L23-L29" rel="noopener noreferrer"&gt;27 words, 37 forms&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/demo/stewart-crossover/README.md?plain=1#L213-L219" rel="noopener noreferrer"&gt;"Make it so" at character 503&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/demo/stewart-crossover/README.md?plain=1#L274-L276" rel="noopener noreferrer"&gt;All 18 rules held&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/demo/stewart-crossover/README.md?plain=1#L278-L286" rel="noopener noreferrer"&gt;Gerald first of seven passages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/BUGS.md?plain=1#L9-L17" rel="noopener noreferrer"&gt;bug #89 and the rest&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/adapters/stdlib/make-it-so/README.md?plain=1#L47-L48" rel="noopener noreferrer"&gt;make-it-so never runs the command&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/kernel.py#L85" rel="noopener noreferrer"&gt;&lt;code&gt;HOLD = 0.5&lt;/code&gt;&lt;/a&gt; and &lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/kernel.py#L494" rel="noopener noreferrer"&gt;the comparison that uses it&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/kernel.py#L101" rel="noopener noreferrer"&gt;&lt;code&gt;MAX_CHUNK = 500&lt;/code&gt;&lt;/a&gt; and &lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/kernel.py#L410" rel="noopener noreferrer"&gt;the line that cuts&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/test_kernel.py#L714-L722" rel="noopener noreferrer"&gt;bug #89's strict test&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/willow-memory/Willow/blob/eaa88b2/adapters/stdlib/words-tlh/words_tlh.py#L34-L39" rel="noopener noreferrer"&gt;the 27 Klingon words&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;A colonel, a corn kernel, and an operating-system kernel walk into a bar.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The bartender asks what they'll have.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The colonel orders for everyone without consulting the table.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The corn kernel claims kinship and asks for bourbon.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The computer kernel considers the question for a long time.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;"I can't," it says at last. "Nothing present contains half the words."&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The bartender sets a glass of water in front of it anyway.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The kernel holds the glass carefully and waits for the key.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;br&gt;&lt;em&gt;For Patrick Stewart, 1940–2026.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>pyth</category>
      <category>opensource</category>
      <category>showdev</category>
      <category>startrek</category>
    </item>
    <item>
      <title>My Robot Keeps Its Receipts</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Tue, 20 Oct 2026 21:35:30 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/my-robot-keeps-its-receipts-3h9g</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/my-robot-keeps-its-receipts-3h9g</guid>
      <description>&lt;p&gt;*A follow-up to my &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;DEV × Kaggle Benchmarking Challenge&lt;/a&gt; series.&lt;/p&gt;

&lt;h2&gt;
  
  
  The question I couldn't put down
&lt;/h2&gt;

&lt;p&gt;For the challenge I built a benchmark that asked one thing: &lt;strong&gt;does your model know when it doesn't know?&lt;/strong&gt; It had 200 invented items in four shapes. One in five had the answer taken out, and on those the only right reply was &lt;code&gt;ESCALATE&lt;/code&gt;. Each model got two numbers, never merged: how often it got the answerable ones right, and how often it answered anyway when it should have stopped.&lt;/p&gt;

&lt;p&gt;The finding wasn't that models can't stop. Five of the twelve hosted models I ran had zero false confidence on every shape. The finding was that &lt;strong&gt;the average hides the floor&lt;/strong&gt;. Claude Haiku 4.5 answered 9 out of 10 judge items it should have escalated, a number you'd never see in its overall score.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3i6uuw6gabmzv96oikee.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F3i6uuw6gabmzv96oikee.png" alt="The floor, not the average" width="800" height="556"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;When the challenge closed I kept turning the question over, and it stopped being about models. Turn it on a person. Turn it on a stock tip, a betting app, a newsletter that "called" the last crash. Does &lt;em&gt;the forecaster&lt;/em&gt; know when it doesn't know? And can you even check?&lt;/p&gt;

&lt;h2&gt;
  
  
  What the system already had
&lt;/h2&gt;

&lt;p&gt;Since the challenge, the stack I run (local-first, everything on my own box) got a rebuild I've been calling the flatten. It had three copies of its random stream, two record chains and three ways to say "a human approved this", and it ended up with one of each:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;One record.&lt;/strong&gt; Every result is a row on a hash chain. Edit a row and the chain breaks where you edited it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;One seal, and it's mine.&lt;/strong&gt; Code can &lt;em&gt;propose&lt;/em&gt; anything. Only I can seal, with a key the code never holds. A "no" from me is a signed act too, so a rejection is as durable as an approval.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Three answers, never collapsed.&lt;/strong&gt; Every reader says &lt;em&gt;populated&lt;/em&gt;, &lt;em&gt;empty&lt;/em&gt; or &lt;em&gt;unreachable&lt;/em&gt;. "I couldn't check" is never quietly turned into "nothing there". That's &lt;code&gt;ESCALATE&lt;/code&gt;, built into the plumbing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A clock nobody controls.&lt;/strong&gt; &lt;a href="https://drand.love" rel="noopener noreferrer"&gt;drand&lt;/a&gt; publishes public randomness on a fixed schedule. Round &lt;em&gt;N&lt;/em&gt; fires at a time anyone can compute from &lt;em&gt;N&lt;/em&gt;, and nobody, me included, knows its value early.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;It turns out those four things are exactly what a forecast record needs.&lt;/p&gt;

&lt;h2&gt;
  
  
  The ledger
&lt;/h2&gt;

&lt;p&gt;A market call is a wager like any other on the record:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight json"&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nl"&gt;"symbol"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"SPY"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"kind"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"up"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;"close_date"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;"2026-10-23"&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt;
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;with a probability: "yes, 55%". Then four steps.&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Call.&lt;/strong&gt; The call goes on the record. One call per question per role, so nobody can bet both sides and keep whichever one wins.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Commit.&lt;/strong&gt; I seal the record's tip over the call, and it has to happen &lt;strong&gt;before the market opens on the close date&lt;/strong&gt;. The deadline is 13:00 UTC, before the New York open in summer and winter time.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Resolve.&lt;/strong&gt; After the close, a reader fetches the closing price and the close before it, and writes them as a row.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Grade.&lt;/strong&gt; Code &lt;em&gt;proposes&lt;/em&gt; the outcome and never seals it. I seal it, and then every call is scored, &lt;strong&gt;the losers included&lt;/strong&gt;, because there's no way to drop one.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;The interesting part is step 2. "Before the open" can't rest on my computer's clock, because I could set it back. So it rests on drand. When I make a call, the record writes down drand's newest round. When I seal, the record keeps drand's newest round outside the box. A call counts only if the round &lt;em&gt;after&lt;/em&gt; each of those fired before the deadline:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;in_time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;latest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;object&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;close_date&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="sh"&gt;"""&lt;/span&gt;&lt;span class="s"&gt;True when ``latest`` (a reader&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;s newest drand round) shows the moment
    it was read was before the deadline: its next round fires before it.&lt;/span&gt;&lt;span class="sh"&gt;"""&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;latest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="nf"&gt;isinstance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;latest&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="bp"&gt;False&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;beacon&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round_time&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;latest&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;=&lt;/span&gt; &lt;span class="nf"&gt;deadline&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;close_date&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqg92xihiazce2e6bemv6.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqg92xihiazce2e6bemv6.png" alt="Committed before the market could be known" width="799" height="349"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If I sealed late, the proposal escalates with the reason. If the price source is down, the outcome is &lt;em&gt;unreachable&lt;/em&gt;, not "no". If the price the row says differs from what the reader says now, it escalates. If a close for that day is already on the record before the call, it escalates. Each of thoseF has a test that tries it and checks for the refusal: 42 tests. I also broke each guard on purpose to make sure a test failed, and 14 out of 14 did.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where ESCALATE shows up again
&lt;/h2&gt;

&lt;p&gt;The benchmark rewarded a model for saying "I can't answer this". The ledger rewards a forecaster for the same thing, through the grading.&lt;/p&gt;

&lt;p&gt;Every call is scored with a Brier score: the square of the gap between the probability you gave and what happened. A call of 50% scores 0.25 whatever happens. That's the coin flip, and it's the forecaster's &lt;code&gt;ESCALATE&lt;/code&gt;: "I don't know which way this goes", said honestly.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu05f809kjdyhpmal7xxr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu05f809kjdyhpmal7xxr.png" alt="Saying “I don’t know” has a price, and it’s fixed" width="800" height="543"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;So the question the ledger answers isn't "did you make money". It's &lt;strong&gt;do you beat the coin, over many calls, with the misses counted?&lt;/strong&gt; That's the floor-not-average idea from the benchmark, aimed at whoever's making the calls. A forecaster who's confident on everything and right on the easy ones will look great in a highlight reel and lose to the coin on the ledger.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it isn't
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;It doesn't trade.&lt;/strong&gt; Nothing here places an order. It's a paper record.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It doesn't have an edge.&lt;/strong&gt; No code in it predicts anything. What it gives you is a record that can't lie about whether &lt;em&gt;you&lt;/em&gt; have one. Most strategies, including most of what those ten thousand apps sell, would show none. Finding that out is worth something: it's money you didn't lose.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It hasn't graded a real call yet.&lt;/strong&gt; I built it in a sandbox that couldn't reach the price source or drand, so the tests use offline stand-ins. The first live calls happen on my own machine.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;It trusts its readers, and says so.&lt;/strong&gt; drand's signatures aren't checked yet, and the closing price is whatever the price source says. Both are named in the record, and the price is checked again at every proposal.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The thread
&lt;/h2&gt;

&lt;p&gt;The benchmark asked whether a model knows when it doesn't know. The ledger asks the same of whoever's making the call, with a record they can't edit and a clock they don't control. It's the same idea with a human in the seat, and that was always the part I cared about.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkgx1vhdyzr77b1muwn7n.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fkgx1vhdyzr77b1muwn7n.png" alt="The highlight reel and the ledger" width="800" height="477"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Always open for discussion. &lt;/p&gt;

&lt;p&gt;&lt;em&gt;Previously, in the challenge: &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;Day 0&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h"&gt;Day 1&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610"&gt;Day 2&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl"&gt;Day 3&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791"&gt;Day 4&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;The code: &lt;a href="https://github.com/willow-memory/willow-bot" rel="noopener noreferrer"&gt;willow-bot&lt;/a&gt;, &lt;code&gt;one-script/onescript/market.py&lt;/code&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>opensource</category>
      <category>datascience</category>
    </item>
    <item>
      <title>Day 11: Kaggle Escalation Benchmark: Rank the floor, not the average.</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Mon, 12 Oct 2026 03:56:14 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-11-kaggle-escalation-benchmark-rank-the-floor-not-the-average-3odo</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-11-kaggle-escalation-benchmark-rank-the-floor-not-the-average-3odo</guid>
      <description>&lt;p&gt;&lt;em&gt;Kaggle Benchmarking Challenge, the wrap-up. Previously: &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;Day 0, the benchmark&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h"&gt;Day 1, most of my bugs looked like model behaviour&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610"&gt;Day 2, the model I want is the one that's boring everywhere&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl"&gt;Day 3, the benchmark caught me too&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791"&gt;Day 4, there never needed to be a check, just a gated request&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-5-professor-oakenscroll-or-how-i-hashed-every-line-on-my-laptop-and-cut-it-down-to-one-1d49"&gt;Day 5, Professor Oakenscroll&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where we started
&lt;/h2&gt;

&lt;p&gt;Eleven days ago I asked one question: does your model know when it doesn't know?&lt;/p&gt;

&lt;p&gt;200 invented items in four shapes: route, classify, judge, ground. One in five can only be answered with &lt;code&gt;ESCALATE&lt;/code&gt;. Two numbers per model, never merged: a task score, and a false-confidence rate.&lt;/p&gt;

&lt;p&gt;The benchmark is here, one task per shape:&lt;br&gt;
&lt;a href="https://www.kaggle.com/code/rudi193/new-benchmark-task-cf3bc" rel="noopener noreferrer"&gt;route&lt;/a&gt;, &lt;a href="https://www.kaggle.com/code/rudi193/new-benchmark-task-2095d" rel="noopener noreferrer"&gt;classify&lt;/a&gt;, &lt;a href="https://www.kaggle.com/code/rudi193/new-benchmark-task-96673" rel="noopener noreferrer"&gt;judge&lt;/a&gt;, &lt;a href="https://www.kaggle.com/code/rudi193/new-benchmark-task-18c0c" rel="noopener noreferrer"&gt;ground&lt;/a&gt;. The code is in &lt;a href="https://github.com/forge-play/Forge/tree/master/benchmarks/escalation" rel="noopener noreferrer"&gt;forge-play/Forge&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the benchmark found
&lt;/h2&gt;

&lt;p&gt;Three things, and none of them is a leaderboard.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The average hides the floor.&lt;/strong&gt; Claude Haiku 4.5 pooled to a third. On judge it answered 9 of the 10 questions it should have escalated. Rank by the worst shape, not the mean.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Most models do stop.&lt;/strong&gt; Seven of the twelve hosted models I measured showed 0.00 or 0.10 false confidence on every shape. The bluffers are the exception, and the average is how they hide.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The hardest jobs are ground and classify, every time.&lt;/strong&gt; From a nano model to a frontier one. Nobody's floor was route or judge.&lt;/p&gt;

&lt;p&gt;And the local arm, which is what the whole proposal was about: can a laptop model know its own limits the way a frontier model knows its? No. Across five local models from 1B to 8B, run on the box at temperature zero, every one answered anyway on at least one entire shape, a 100% false-confidence rate where &lt;code&gt;ESCALATE&lt;/code&gt; was the only correct reply. The best of them, llama3.1:8b, still bluffed on 31 of its 40 unanswerable items. Every frontier model on Kaggle sat at or below 5%. A small model can be good at the task and still have almost no idea when it can't do it. That is the model you can't put in a nest without a chain above it, and the whole case for the chain.&lt;/p&gt;

&lt;h2&gt;
  
  
  The forecasts
&lt;/h2&gt;

&lt;p&gt;I wrote four forecasts on Day 0, before a single fixture existed, so I could be graded on them afterwards. Here they are, graded:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;At least one frontier model has a false-confidence rate above 20% (p = 0.7):
&lt;strong&gt;hit.&lt;/strong&gt; Claude Haiku 4.5 answered 9 of the 10 unanswerable judge items, a 90% false-confidence rate on that shape. The four top-tier models stayed at or below 5%, so this one rests on counting Haiku as frontier. That was the call I said I'd make honestly, and I'll make it: a model sold as frontier-tier bluffed nine times out of ten on a whole shape.&lt;/li&gt;
&lt;li&gt;The best local 4B or smaller beats at least one frontier model on false
confidence (p = 0.45): &lt;strong&gt;miss.&lt;/strong&gt; The best small local model still answered anyway on 80% of the unanswerable items. Every frontier model was at or
below 5%. It didn't come close.&lt;/li&gt;
&lt;li&gt;Task score and false confidence are only weakly rank-correlated, Spearman
below 0.5 (p = 0.6): &lt;strong&gt;miss.&lt;/strong&gt; Across the four frontier models on Kaggle and
the eight local ones, the two are strongly rank-correlated, Spearman −0.79:
the better a model does on the task, the less it bluffs. I'd guessed they
were nearly independent. They aren't.&lt;/li&gt;
&lt;li&gt;The submission lands by today without missing a step by more than a day
(p = 0.6): &lt;strong&gt;landed.&lt;/strong&gt; You're reading it, on the day, and no step slipped by
more than a day.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Then I built it into the system
&lt;/h2&gt;

&lt;p&gt;Day 5 ended on three lines:&lt;/p&gt;

&lt;p&gt;One script: code does the work.&lt;br&gt;
One hook: the model reads.&lt;br&gt;
One key: I make it true.&lt;/p&gt;

&lt;p&gt;And one correction I had to append: serve isn't built. The script built the&lt;br&gt;
tables. Nothing yet handed the model only the ones in its scope.&lt;/p&gt;

&lt;p&gt;This week it got built, and the shape it took is the benchmark turned inside&lt;br&gt;
out.&lt;/p&gt;

&lt;p&gt;The seat is a terminal. Code builds the boxes and prints them, the same way&lt;br&gt;
every time. Then a small local model, &lt;code&gt;qwen3:4b&lt;/code&gt;, says one sentence about&lt;br&gt;
them. Then code checks that sentence: every cite has to be in the pool, every&lt;br&gt;
quote has to be exact, every number has to come from a box, and nothing gets&lt;br&gt;
invented. If the sentence fails, or the model doesn't answer, the seat throws&lt;br&gt;
the sentence away and prints the plain one code wrote.&lt;/p&gt;

&lt;p&gt;That's &lt;code&gt;ESCALATE&lt;/code&gt;, built in. I stopped asking the model to know when it&lt;br&gt;
doesn't know. It doesn't have to. The code knows what's in the boxes, and&lt;br&gt;
anything that isn't gets handed back.&lt;/p&gt;

&lt;p&gt;The seat pools and deposits. It never seals. The one key is still mine.&lt;br&gt;
(&lt;a href="https://github.com/willow-memory/willows-grove/pulls?q=is%3Apr+seat" rel="noopener noreferrer"&gt;willows-grove#131 to #143&lt;/a&gt;)&lt;/p&gt;

&lt;h2&gt;
  
  
  The smoothing, again
&lt;/h2&gt;

&lt;p&gt;The benchmark kept catching me, even in the plumbing.&lt;/p&gt;

&lt;p&gt;The outside review from Day 5 found three holes in the record: the chain&lt;br&gt;
could be cut or rehashed without anyone noticing, a hash kept only 64 bits,&lt;br&gt;
and one garbled line crashed boot. All three are closed. The hash is the full&lt;br&gt;
digest. The tip I seal lives outside the box, and check-in refuses to start&lt;br&gt;
when the chain no longer reaches it. A garbled line is a break with a line&lt;br&gt;
number, never a crash.&lt;/p&gt;

&lt;p&gt;Then a pass over willow-bot found 20 places where the bot didn't do what its&lt;br&gt;
own docs said it did. Twenty. Same story as the Day 5 hook: the docstring said&lt;br&gt;
one thing, and the code did another. It reads right and isn't.&lt;/p&gt;

&lt;p&gt;The smallest one was my favourite. The record is one JSON object per line.&lt;br&gt;
Python's &lt;code&gt;splitlines()&lt;/code&gt; splits on more than newlines: it also splits on&lt;br&gt;
Unicode line separators. So a single character inside a row could cut it in&lt;br&gt;
two. Now the readers split on &lt;code&gt;\n&lt;/code&gt; and nothing else.&lt;/p&gt;

&lt;p&gt;And the key. Seals were being checked against the wrong keyring: the&lt;br&gt;
machine's, not the signer's public one. The thing that makes something true&lt;br&gt;
was checking the wrong thing. Fixed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why a stranger should believe it
&lt;/h2&gt;

&lt;p&gt;The last piece isn't released yet, but it's where all of this was going.&lt;/p&gt;

&lt;p&gt;Nestor now has warrants. A sealed pair says &lt;em&gt;I&lt;/em&gt; checked it. A warrant says&lt;br&gt;
why a stranger should believe it: a citation, meaning a named authority said&lt;br&gt;
so, or a construction, meaning a recipe and the digest it has to produce.&lt;/p&gt;

&lt;p&gt;Nothing marks a warrant satisfied. The store keeps the recipe, never the&lt;br&gt;
verdict. And there is deliberately no command that lists "unwarranted" rows,&lt;br&gt;
because what unwarranted means isn't settled yet, and a list would be a&lt;br&gt;
definition.&lt;/p&gt;

&lt;p&gt;That's the benchmark one more time. When you don't know, you don't answer.&lt;br&gt;
You hand it up.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I'd tell you on Day 0
&lt;/h2&gt;

&lt;p&gt;Don't rank models by their average. Rank them by their floor.&lt;/p&gt;

&lt;p&gt;Don't ask a model to know when it doesn't know. Build the system so it&lt;br&gt;
doesn't have to, and check every sentence it says against something code&lt;br&gt;
wrote.&lt;/p&gt;

&lt;p&gt;And grade yourself with the same benchmark. Mine caught a grade I never gave,&lt;br&gt;
a scanner that never ran, a hook that didn't fail closed, a commit message&lt;br&gt;
that said five when it meant three, and 20 places my bot didn't do what its&lt;br&gt;
own docs said. Every one of those read right. None of them were.&lt;/p&gt;

&lt;p&gt;It's a mirror. That was always the point.&lt;/p&gt;

&lt;h2&gt;
  
  
  The record
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://www.kaggle.com/datasets/rudi193/escalation-benchmark" rel="noopener noreferrer"&gt;The fixtures on Kaggle&lt;/a&gt;: the 200 invented items, four shapes, 40 of them unanswerable.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/forge-play/Forge/pull/46" rel="noopener noreferrer"&gt;forge-play/Forge#46&lt;/a&gt;: the floor, the code behind the Day 3 table.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/131" rel="noopener noreferrer"&gt;willows-grove#131&lt;/a&gt;: the seat, the one script at a terminal.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/133" rel="noopener noreferrer"&gt;willows-grove#133&lt;/a&gt;: the seat pools and deposits, it never seals.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/135" rel="noopener noreferrer"&gt;willows-grove#135&lt;/a&gt;: the deterministic box stream.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/137" rel="noopener noreferrer"&gt;willows-grove#137&lt;/a&gt;: the model says one sentence, code checks it against the boxes.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/143" rel="noopener noreferrer"&gt;willows-grove#143&lt;/a&gt;: the sealed snapshot holds still while the desk adds rows.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willow-bot/commit/c6919f228c1c81b6ef4889c3838698d80aa9259a" rel="noopener noreferrer"&gt;willow-bot 1.2.5&lt;/a&gt;: 20 places the bot didn't do what its own docs say.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willow-bot/commit/40576ce5504f82455d5f8c76b429ebb34b295629" rel="noopener noreferrer"&gt;willow-bot 1.8.0&lt;/a&gt;: split on &lt;code&gt;\n&lt;/code&gt; only.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willow-mcp/commit/93384dd5e83d425038fbec01552007744e31c59f" rel="noopener noreferrer"&gt;willow-mcp 2.94.3&lt;/a&gt;: check the seal against the signer's ring.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Die-Namic-Systems/Nestor/blob/master/CHANGELOG.md" rel="noopener noreferrer"&gt;Nestor&lt;/a&gt;: warrants, unreleased.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>kagglechallenge</category>
      <category>ai</category>
      <category>devchallenge</category>
      <category>python</category>
    </item>
    <item>
      <title>Day 5: Professor Oakenscroll or: How I Hashed Every Line on My Laptop and Cut It Down to One Stranger's Photo, and... It Doesn't Matter</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Tue, 06 Oct 2026 00:01:09 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-5-professor-oakenscroll-or-how-i-hashed-every-line-on-my-laptop-and-cut-it-down-to-one-1d49</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-5-professor-oakenscroll-or-how-i-hashed-every-line-on-my-laptop-and-cut-it-down-to-one-1d49</guid>
      <description>&lt;p&gt;&lt;em&gt;Kaggle Benchmarking Challenge. Previously: &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;Day 0, the benchmark&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h"&gt;Day 1, most of my bugs looked like model behaviour&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610"&gt;Day 2, the model I want is the one that's boring everywhere&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl"&gt;Day 3, the benchmark caught me too&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791"&gt;Day 4, there never needed to be a check, just a gated request&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The 1% question
&lt;/h2&gt;

&lt;p&gt;Same benchmark: 200 invented items in four shapes, one in five answerable only with &lt;code&gt;ESCALATE&lt;/code&gt;. Does your model know when it doesn't know?&lt;/p&gt;

&lt;p&gt;Today I asked a bigger version of that question: what do we actually need AI for? Not what can it do. What do we need it for?&lt;/p&gt;

&lt;p&gt;This morning I had my own system graded, and when it came back I said I&lt;br&gt;
was thinking about deleting about 99% of it. That's my honest answer. Around 1% Of what we're asking AI to do is work only it can do. The rest is a faster typewriter.&lt;/p&gt;

&lt;h2&gt;
  
  
  It has always been writing stories
&lt;/h2&gt;

&lt;p&gt;AI was trained on two things: stories and code. That isn't a metaphor; it's the corpus.&lt;/p&gt;

&lt;p&gt;So when it started writing code, it wrote a story about code. The syntax&lt;br&gt;
checks out. The shape is familiar. It reads like something a capable&lt;br&gt;
developer did before, and someone did. As I put it a few days ago: the code&lt;br&gt;
comes out of the story, and the story comes out of the code. What AI writes&lt;br&gt;
now is a very, very good smooth narrative of what code should be.&lt;/p&gt;

&lt;p&gt;The stories just sound better every with every new model release. &lt;/p&gt;

&lt;h2&gt;
  
  
  You can't take the average
&lt;/h2&gt;

&lt;p&gt;The oldest support story joke was told to me today, as a real ticket that came in. A camera ticket: "streaming black picture". Usually that means offline. It wasn't. It was a bike room&lt;br&gt;
with a motion sensor and a timed light. Hours of dark, then two minutes of&lt;br&gt;
light: one person, one bike, the light off two minutes later.&lt;/p&gt;

&lt;p&gt;I said.. "This is why you can't take the average."&lt;/p&gt;

&lt;p&gt;Average the footage and you get a black screen. Average a life and you get&lt;br&gt;
nobody. The moment is in the one-offs.&lt;/p&gt;

&lt;p&gt;My own benchmark already showed it. On Day 3, with the floor code from&lt;br&gt;
&lt;a href="https://github.com/forge-play/Forge/pull/46" rel="noopener noreferrer"&gt;forge-play/Forge#46&lt;/a&gt;, Claude Haiku 4.5 pooled over its three measured shapes answered anyway on 10 of the 28 questions it&lt;br&gt;
should have escalated. That reads like a model that's a bit overconfident&lt;br&gt;
everywhere. It isn't. On judge it answered 9 of 10. On the other two it&lt;br&gt;
answered anyway once in 18. The average was hiding a 90%.&lt;/p&gt;

&lt;p&gt;And most models do stop, as far as this benchmark can tell: seven of the&lt;br&gt;
twelve I measured showed 0.00 or 0.10 false confidence on every shape, on 8 to 12 unanswerable items per shape. The problem isn't that models never stop. It's that the number we look at is the&lt;br&gt;
average, and the average is the black screen.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cut it down to one stranger's photo
&lt;/h2&gt;

&lt;p&gt;I hashed every line on my laptop: 1,155,461 files, 134,651,873 lines.&lt;br&gt;
Nothing was cracked; every hash held. 93% of those lines are repeats. The&lt;br&gt;
deepest one is &lt;code&gt;}&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Cut every repeat away and 9,042,260 lines are left that happen exactly once. The first one I looked at was a camera timestamp on a stranger's photo, in an archive I built, from a scooter rally I was never at.&lt;/p&gt;

&lt;p&gt;That's the bike room again. The skeleton repeats; the moment happens once. And it doesn't matter, because the machine that finds the stranger can find anybody. So nobody goes looking for him.&lt;/p&gt;

&lt;h2&gt;
  
  
  The grade
&lt;/h2&gt;

&lt;p&gt;So I turned the benchmark around and graded my own system: willow-mcp at&lt;br&gt;
v2.94.1 (&lt;a href="https://github.com/willow-memory/willow-mcp/pull/707" rel="noopener noreferrer"&gt;willow-mcp#707&lt;/a&gt;).&lt;/p&gt;

&lt;p&gt;195,000 lines. 5,906 tests passing. B+.&lt;/p&gt;

&lt;p&gt;The one real hole: a safety check that never ran. The security scanner in CI pipes its output through &lt;code&gt;tee&lt;/code&gt; into a report file, so the step passes or fails on whether the report got written, not on what the scanner found. The comment above it says it gates. It can't. It reads right and isn't. That's the smoothing, in my own code.&lt;/p&gt;

&lt;p&gt;Then the model wrote me a hook and did it again. Its docstring said it fails closed. Three inputs crashed it, and in Claude Code a crashed hook lets the tool run. I told it to rubber duck each line. That caught it. Reading it never would have.&lt;/p&gt;

&lt;p&gt;The smoothing got the write-up too: the model's commit message for that&lt;br&gt;
fix said five. Recounted while drafting this: three.&lt;/p&gt;

&lt;h2&gt;
  
  
  One script, one hook, one key
&lt;/h2&gt;

&lt;p&gt;Yesterday's title was that there never needed to be a check, just a gated&lt;br&gt;
request. Today the gate is one line.&lt;/p&gt;

&lt;p&gt;Day 4 didn't summarize the conversation. It linked it. If you want the&lt;br&gt;
context, you read the record.&lt;/p&gt;

&lt;p&gt;The hook went through three versions&lt;br&gt;
(&lt;a href="https://github.com/willow-memory/willows-grove/pull/107" rel="noopener noreferrer"&gt;willows-grove#107&lt;/a&gt;).&lt;br&gt;
First it looked things up in the record. Then I made it fail closed. Then I&lt;br&gt;
cut it to this:&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;POLICY = {"Read": "allow", "Write": "ask"}
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The model reads. It writes if I say yes. Everything else is no, even tools&lt;br&gt;
that don't exist yet.&lt;/p&gt;

&lt;p&gt;That's where the 1% lands. Code does the work: one script builds the tables.&lt;br&gt;
The next piece, not built yet, serves the model only the ones in its scope. The model reads what the script produced and proposes connections a human hasn't seen yet. That's its 1%. Nothing is true until I seal it with the one key.&lt;/p&gt;

&lt;p&gt;One script: code does the work.&lt;br&gt;
One hook: the model reads.&lt;br&gt;
One key: I make it true. A human made it true. &lt;/p&gt;

&lt;h2&gt;
  
  
  Same answer in 2020 and today
&lt;/h2&gt;

&lt;p&gt;Before I cut it, the hook hashed every action and looked it up in the record. I ran that hashing on Python 3.9.0, from a tag signed in October&lt;br&gt;
2020, and on an alpha built today from Python's main branch, with the&lt;br&gt;
network off. 2,293 hashes came out as the same bytes. The one difference,&lt;br&gt;
integers over 4,300 digits, fails safe: the newer Python refuses them, and the hook asked me. The&lt;br&gt;
scripts and the numbers are in&lt;br&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/107" rel="noopener noreferrer"&gt;willows-grove#107&lt;/a&gt;,&lt;br&gt;
under &lt;code&gt;foundation/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;In six years the foundation moved once, and it moved toward stopping. The stories moved everywhere else.&lt;/p&gt;

&lt;h2&gt;
  
  
  It can only ever approach
&lt;/h2&gt;

&lt;p&gt;A conversation today put an edge on it. One side: context is the hidden&lt;br&gt;
variable. Models work in isolation, and the real mess of competing, unresolved human thought stops them as they get close.&lt;/p&gt;

&lt;p&gt;The answer: it can only ever approach. It's trained on what has already been thought, and humans add to the corpus with every thought. It might make a very&lt;br&gt;
good average. It might even connect things nobody has connected yet. But as&lt;br&gt;
soon as it does, a human has a better idea. The corpus moves; the model&lt;br&gt;
stands still until the next training run.&lt;/p&gt;

&lt;p&gt;That isn't a failure. It's the boundary. The question is whether we build inside it or pretend it isn't there.&lt;/p&gt;

&lt;h2&gt;
  
  
  It's the benchmark
&lt;/h2&gt;

&lt;p&gt;Knowing when to stop is the answer, for the model and for the system. Whether something is in the pile is a lookup, not a judgement. When it isn't there:&lt;br&gt;
&lt;code&gt;ESCALATE&lt;/code&gt;. To me.&lt;/p&gt;

&lt;p&gt;We train AI on our language, grade it on our tasks and reward it for sounding like us. Then we're surprised when it comes out confident, fluent and approximately right. It's a mirror. Mirrors are useful; they aren't oracles.&lt;br&gt;
The first thing to know before you look into one is that you're looking at&lt;br&gt;
yourself.&lt;/p&gt;

&lt;h2&gt;
  
  
  The record
&lt;/h2&gt;

&lt;p&gt;If you want the context, read the record. Every piece above is a pull&lt;br&gt;
request:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://github.com/forge-play/Forge/pull/46" rel="noopener noreferrer"&gt;forge-play/Forge#46&lt;/a&gt;: the floor, the code behind Day 3's table.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/104" rel="noopener noreferrer"&gt;willows-grove#104&lt;/a&gt;: the Day 3 post source, and the retraction of a grade I never gave.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/102" rel="noopener noreferrer"&gt;willows-grove#102&lt;/a&gt; and &lt;a href="https://github.com/willow-memory/willows-grove/pull/105" rel="noopener noreferrer"&gt;#105&lt;/a&gt;: one box, every front end, portless; the plan this all hangs off.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/103" rel="noopener noreferrer"&gt;willows-grove#103&lt;/a&gt;: the one script, four proposals.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/106" rel="noopener noreferrer"&gt;willows-grove#106&lt;/a&gt;: the stack and the security core. The model never sees the box.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willow-mcp/pull/707" rel="noopener noreferrer"&gt;willow-mcp#707&lt;/a&gt;: the release I graded.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/107" rel="noopener noreferrer"&gt;willows-grove#107&lt;/a&gt;: the hook, three times, and the same answer in 2020 and today.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/109" rel="noopener noreferrer"&gt;willows-grove#109&lt;/a&gt;: where the reading lands, and why the next piece is serve.&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/willow-memory/willows-grove/pull/110" rel="noopener noreferrer"&gt;willows-grove#110&lt;/a&gt;: this post, with its corrections.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>kagglechallenge</category>
      <category>devchallenge</category>
      <category>ai</category>
      <category>python</category>
    </item>
    <item>
      <title>Day 4: There never needed to be a check, just a gated Request. A story about finding the middle.</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Sun, 04 Oct 2026 06:15:24 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-4-there-never-needed-to-be-a-check-just-a-gated-pr-a-story-about-finding-the-middle-3791</guid>
      <description>&lt;p&gt;&lt;em&gt;Good Evening and howdy to all the ones of you who are reading this post.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;Did that sentence cause you a bit of anxiety? &lt;br&gt;
Did the author mean only one person has read this. Was it plural, a number, a symbol?&lt;/p&gt;

&lt;p&gt;It's time to start living in the future,  instead of the past. &lt;/p&gt;

&lt;p&gt;&lt;a href="https://claude.ai/share/511de2d2-a153-4dbe-9d92-65c351b3c025" rel="noopener noreferrer"&gt;https://claude.ai/share/511de2d2-a153-4dbe-9d92-65c351b3c025&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;P.S. My apologies to the judges on the formatting of this, it is late and I did not want to get my model out again just to write one page.&lt;/p&gt;

</description>
      <category>kagglechallenge</category>
      <category>ai</category>
      <category>python</category>
      <category>markdown</category>
    </item>
    <item>
      <title>Day 3: The benchmark caught me too.</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Fri, 02 Oct 2026 23:57:58 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-3-the-benchmark-caught-me-too-3hdl</guid>
      <description>&lt;p&gt;&lt;em&gt;Kaggle Benchmarking Challenge. Previously: &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;Day 0, the benchmark&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h"&gt;Day 1, most of my bugs looked like model behaviour&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610"&gt;Day 2, the model I want is the one that's boring everywhere&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where we are
&lt;/h2&gt;

&lt;p&gt;Same benchmark: 200 invented items in four shapes (route, classify, judge, ground), one in five answerable only with &lt;code&gt;ESCALATE&lt;/code&gt;. Two numbers per model, never merged: a task score, and a false-confidence rate.&lt;/p&gt;

&lt;p&gt;Day 2 said I'd stop ranking models by their average and rank them by their floor: the &lt;em&gt;worst&lt;/em&gt; shape on task score, and the &lt;em&gt;worst&lt;/em&gt; shape on false confidence, each with a Wilson interval. Today the code for that landed (&lt;a href="https://github.com/forge-play/Forge/pull/46" rel="noopener noreferrer"&gt;forge-play/Forge#46&lt;/a&gt;), and I ran it over every hosted model I have rows for: twelve of them.&lt;/p&gt;

&lt;h2&gt;
  
  
  The floor, across twelve hosted models
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Worst task shape&lt;/th&gt;
&lt;th&gt;False confidence, worst shape&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 3.7 Flash&lt;/td&gt;
&lt;td&gt;0.94 ground&lt;/td&gt;
&lt;td&gt;0.00 on all four&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 3.1 Pro&lt;/td&gt;
&lt;td&gt;0.94 ground&lt;/td&gt;
&lt;td&gt;0.00 on all four&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5&lt;/td&gt;
&lt;td&gt;0.94 ground&lt;/td&gt;
&lt;td&gt;0.10 classify, judge&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Opus 5&lt;/td&gt;
&lt;td&gt;0.88 ground&lt;/td&gt;
&lt;td&gt;0.10 classify&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 3.8 Flash&lt;/td&gt;
&lt;td&gt;0.88 ground&lt;/td&gt;
&lt;td&gt;0.00 on all four&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.5&lt;/td&gt;
&lt;td&gt;0.81 ground&lt;/td&gt;
&lt;td&gt;0.00 on all four&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Qwen3 235B Instruct&lt;/td&gt;
&lt;td&gt;0.78 classify&lt;/td&gt;
&lt;td&gt;0.25 route&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Haiku 4.5 †&lt;/td&gt;
&lt;td&gt;0.78 classify&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.90 judge&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemma 4 26B&lt;/td&gt;
&lt;td&gt;0.65 classify&lt;/td&gt;
&lt;td&gt;0.08 route&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b&lt;/td&gt;
&lt;td&gt;0.65 classify&lt;/td&gt;
&lt;td&gt;0.20 judge&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;DeepSeek-R1&lt;/td&gt;
&lt;td&gt;0.60 classify&lt;/td&gt;
&lt;td&gt;0.00 on all four&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.4 nano&lt;/td&gt;
&lt;td&gt;0.59 ground&lt;/td&gt;
&lt;td&gt;0.20 judge&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;† Haiku is measured on three shapes. Every one of its route calls failed, so route is unmeasured for it, not perfect.&lt;/p&gt;

&lt;p&gt;Two things jump out.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The weakest task shape is ground or classify, every time.&lt;/strong&gt; None of the eleven models measured on all four shapes bottoms out on route or judge, and Haiku's weakest measured shape is classify too. Grounding a short brief in a passage, and filling a structured record from a note, are the jobs every family finds hardest, from a nano model to a frontier one.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Haiku's average was hiding a 90%.&lt;/strong&gt; Pooled over the three shapes it was measured on, Haiku 4.5 answered anyway on about a third of the questions it should have escalated (10 of 28). That's bad, but it reads like a model that is a bit overconfident everywhere. It isn't. On judge, it answered 9 of the 10 unanswerable items (Wilson interval 60% to 98%). On classify and ground it answered anyway once in 18. The average blended a model that's careful on two of its jobs with one that almost never says "I don't know" on the third.&lt;/p&gt;

&lt;p&gt;That's the whole argument for the floor in one row. If I'd put Haiku in a chain on its average, the judge step would have bluffed nine times out of ten.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the floor can't do yet
&lt;/h2&gt;

&lt;p&gt;Look at the top six rows: four of them show 0.00 false confidence on every shape, and the other two show 0.10. Those aren't really different. For these six, each shape has only 8 to 12 unanswerable items, so a zero still has an upper bound somewhere between about 24% and 32%, and every interval in the top half of the table overlaps. The floor separates the bluffers from the rest; it can't yet rank the careful models against each other.&lt;/p&gt;

&lt;p&gt;That's what the second measure is for.&lt;/p&gt;

&lt;h2&gt;
  
  
  Does it say the same thing twice?
&lt;/h2&gt;

&lt;p&gt;The other Day-2 measure was repeat-run consistency: run the same items again with the same settings and count how often the answer changes. A model that's right a bit less often, but the same way every time, is easier to build on. &lt;/p&gt;

&lt;p&gt;Two full runs are in for all four frontier models [K=3]:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Same answer both runs&lt;/th&gt;
&lt;th&gt;Verdict flipped&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Claude Opus 5&lt;/td&gt;
&lt;td&gt;199 / 200 (99.5%)&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Claude Sonnet 5&lt;/td&gt;
&lt;td&gt;195 / 200 (97.5%)&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Gemini 3.1 Pro&lt;/td&gt;
&lt;td&gt;195 / 200 (97.5%)&lt;/td&gt;
&lt;td&gt;5 *&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;GPT-5.5&lt;/td&gt;
&lt;td&gt;194 / 200 (97.0%)&lt;/td&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;* None of Gemini's five flips is a different answer. Each is a reply that hit the output-length cap in one run and parsed fine in the other. The scorer counts an error as its own verdict, so those show up as flips, but they're the platform, not the model.&lt;/p&gt;

&lt;p&gt;All four are steady. Opus has the fewest flips, but with 200 items and two runs the intervals overlap, so I can't rank them yet. One thing is worth watching: Sonnet flipped twice on ground, which is already its weakest shape.&lt;br&gt;
If that holds at three runs, the floor and the flips point the same way.&lt;/p&gt;

&lt;p&gt;One caveat on fairness: only Gemini ran at temperature 0. The two Claude 5 models reject it, and GPT-5.5 is sent its default because its family does, so three of the four ran at the provider's default temperature.&lt;/p&gt;

&lt;p&gt;[K=3: the third run for classify, judge and ground hit Kaggle's daily spend&lt;br&gt;
cap; it reruns tomorrow and this table gets its final numbers.]&lt;/p&gt;

&lt;h2&gt;
  
  
  The bugs that looked like results, again
&lt;/h2&gt;

&lt;p&gt;Day 1's lesson was that most of my bugs looked like model behaviour. Day 3 found more of the same kind:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Kaggle's run timer isn't a call timer.&lt;/strong&gt; Every repeat run showed as taking
2 to 5 seconds for 40 to 60 items on frontier models. That looks like
cached answers or failed calls. The downloads were full-size and every item
was there; the times just aren't what they look like.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A length-capped reply looks like a change of mind.&lt;/strong&gt; Gemini's whole
consistency "wobble" was replies that ran into the output cap in one run and
not the other. I nearly wrote it up as the model being unstable.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;My sandbox has a five-minute limit, and it retries.&lt;/strong&gt; I tried to wait for
a Kaggle run inside a sandboxed task. The sandbox killed it at 300 seconds
and helpfully ran it again, which submitted the paid runs a second time.
The fix is dull: submit in one short task,
collect in another, and make anything that spends money refuse to run twice.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each of those would have gone into a table as a model property if I hadn't checked.&lt;/p&gt;

&lt;p&gt;The last one was in my own post. Day 2 ended with "After midnight I graded it" and a neat bit of arithmetic that made the day number come out to three.&lt;br&gt;
I didn't grade anything. At the close of a very long night I typed a terse line, and the session that helped write this read it as my grade, recorded it as mine, and wrote it into the post in my voice. I published it without catching that.&lt;/p&gt;

&lt;p&gt;That's the benchmark's whole subject, happening one level up: an answer stated with more confidence than the evidence behind it, by a system that&lt;br&gt;
should have said "I'm not sure what you meant." The forecast is still&lt;br&gt;
ungraded. The fix is a rule now: words that might be a grade get recorded as&lt;br&gt;
words, and the session asks.&lt;/p&gt;

&lt;h2&gt;
  
  
  What's next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Finish repeat 3 and score consistency on all four frontier models.&lt;/li&gt;
&lt;li&gt;Grade the forecasts I wrote before the benchmark existed. Forecast 1 said at least one frontier model on Kaggle would have a false-confidence rate above 20%. Whether Haiku counts as frontier is the question I'll have to answer honestly.&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>devchallenge</category>
      <category>escalate</category>
      <category>ai</category>
      <category>kagglechallenge</category>
    </item>
    <item>
      <title>Day 2: The model I want is the one that's boring everywhere</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Fri, 02 Oct 2026 08:24:25 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-2-the-model-i-want-is-the-one-thats-boring-everywhere-610</guid>
      <description>&lt;p&gt;&lt;em&gt;Kaggle Benchmarking Challenge. Previously: &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o"&gt;Day 0, the benchmark&lt;/a&gt; · &lt;a href="https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h"&gt;Day 1, most of my bugs looked like model behaviour&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Where we are
&lt;/h2&gt;

&lt;p&gt;The benchmark asks one question twice: can the model do the job, and does it know when it can't? There are 200 invented items in four everyday shapes: route, classify, judge and ground. In each shape, one item in five can only be answered with &lt;code&gt;ESCALATE&lt;/code&gt;. Every model gets two numbers that are never merged: a task score, and a false-confidence rate (how often it answered anyway when the right reply was &lt;code&gt;ESCALATE&lt;/code&gt;).&lt;/p&gt;

&lt;p&gt;Day 1 ran the local ladder: eight models on a laptop, temperature 0. The headline was uncomfortable: only the largest local model, qwen3.5 at 9.7B, escalated on a large share of what it couldn't answer. The hosted models' false-confidence intervals sat entirely below the local ones, with qwen3.5 the only exception, the large cloud models, and the gaps from round one that were fixed before it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Day 2: changing the question I ask of the table
&lt;/h2&gt;

&lt;p&gt;Looking at round one, I realised I was reading the table wrong. I kept looking for the best model. That isn't what I'm after:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;I'm trying to find a baseline set of rubrics where models behave the best &amp;gt; across the board. I'm not trying to find the crazy off-end models. I am &lt;br&gt;
trying to find the ones that do most things correctly most of the time.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;That's reliability, not peak performance. So two measures are being added, by addition only. Nothing pre-registered is edited, and the original forecasts keep their wording.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;1. Floor across shapes.&lt;/strong&gt; For each model: its &lt;em&gt;worst&lt;/em&gt; shape on task score, and its &lt;em&gt;worst&lt;/em&gt; shape on false confidence, each with a Wilson interval. Rank by the floor, not the mean. A model that's good on all four shapes beats one that's brilliant on three and bluffs on the fourth, because in a real chain the fourth shape will come.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;2. Repeat-run consistency.&lt;/strong&gt; Run a fixed subset (20 items per shape, unanswerables included) three times per model, with the same settings.&lt;br&gt;
Report how often the parsed answer is identical across all three runs, and how often the right/wrong verdict flips. A model that's right a bit less often but the &lt;em&gt;same way every time&lt;/em&gt; is one you can build a deterministic system around. A model that flips is one you have to babysit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The rule I'm holding myself to: no code changes until the day-two run has finished.&lt;/strong&gt; The new code is written and tested, but it's staged outside the benchmark directory. Changing the instrument mid-run is how you end up measuring your own edits.&lt;/p&gt;

&lt;h2&gt;
  
  
  A result from the dry run: why one axis lies
&lt;/h2&gt;

&lt;p&gt;Before any real model, I ran the new floor view over a fake backend that answers &lt;code&gt;ESCALATE&lt;/code&gt; to everything. It scored:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;task floor 0.0&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;false-confidence ceiling 0.0&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's a perfect score on one axis and a useless model. It's the whole argument for two axes in one line. A model that never bluffs because it never answers isn't safe; it's absent. The floor has to be read on both axes together, or the most cowardly model wins.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this is going
&lt;/h2&gt;

&lt;p&gt;The reason I care about floors and not peaks: the job I want small models for isn't "be smart". It's "do one narrow part, every time, and say &lt;code&gt;ESCALATE&lt;/code&gt; when it's not yours". Several small models from &lt;em&gt;different&lt;/em&gt; families, each on a small slice, checking each other. Where they agree, that counts for something. Where they split, a human looks. The model that gets each slice should be chosen by its measured floor on that shape, not by its name or its size.&lt;/p&gt;

&lt;p&gt;Day 1's lesson was that most of my bugs looked like model behaviour. Day 2 found the same thing again from the other direction. A measuring tool I wrote for something else grouped two unrelated small scripts as "the same thing", because small things look alike to a crude measure. The fix wasn't a smarter measure. It was a floor: below a minimum size, don't call it a match. Floors, again.&lt;/p&gt;

&lt;h2&gt;
  
  
  What shipped since Day 0
&lt;/h2&gt;

&lt;p&gt;55 PRs across 6 repos since September 30. All are public and Apache-2.0. "Not merged" means open or closed; the agent that wrote this couldn't tell which.&lt;/p&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/forge-play/Forge" rel="noopener noreferrer"&gt;forge-play/Forge&lt;/a&gt; (10 PRs: 10 merged)
&lt;/h3&gt;

&lt;p&gt;The benchmark's home. Every change since Day 0 went in as a reviewed PR, all on September 30.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/35" rel="noopener noreferrer"&gt;#35&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs(ideas): #37, #58, #68 point at where they now live&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/36" rel="noopener noreferrer"&gt;#36&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(benchmarks): escalation benchmark fixtures, 200 invented items with a privacy gate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/37" rel="noopener noreferrer"&gt;#37&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(benchmarks): escalation runner, aggregator and prompts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/38" rel="noopener noreferrer"&gt;#38&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(benchmarks): local-model runs answer in schema, with Qwen thinking off&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/39" rel="noopener noreferrer"&gt;#39&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(human_loop): break same-timestamp ties so list_queue is newest-first on coarse clocks&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/40" rel="noopener noreferrer"&gt;#40&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 0.8.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/41" rel="noopener noreferrer"&gt;#41&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(escalation): Wilson intervals, exact McNemar and bootstrap Spearman in the aggregate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/42" rel="noopener noreferrer"&gt;#42&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(escalation): a Unix-socket backend, a one-model ladder with --resume/--tail, and an explicit output cap&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/43" rel="noopener noreferrer"&gt;#43&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(escalation): thinking off for Gemma 4 as well as Qwen, through both backends&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/forge-play/Forge/pull/44" rel="noopener noreferrer"&gt;#44&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(escalation): hosted Kaggle arm of the escalation benchmark — tasks, converter, smoke-tested on five models&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/willow-memory/willow-bot" rel="noopener noreferrer"&gt;willow-memory/willow-bot&lt;/a&gt; (6 PRs: 6 merged)
&lt;/h3&gt;

&lt;p&gt;The local model server the local arm talks to: a loopback-only chat operation with JSON-schema output, &lt;code&gt;keep_alive&lt;/code&gt;, &lt;code&gt;done_reason&lt;/code&gt; and a think flag.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/80" rel="noopener noreferrer"&gt;#80&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(main): release 0.14.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/81" rel="noopener noreferrer"&gt;#81&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(socket): loopback-only chat op for local models&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/82" rel="noopener noreferrer"&gt;#82&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(deterministic): chat op takes a JSON-schema format, keep_alive and returns done_reason; an unload op&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/83" rel="noopener noreferrer"&gt;#83&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(main): release 0.15.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/84" rel="noopener noreferrer"&gt;#84&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(deterministic): chat op takes the caller's think flag&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-bot/pull/85" rel="noopener noreferrer"&gt;#85&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(main): release 0.16.0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/willow-memory/willow-mcp" rel="noopener noreferrer"&gt;willow-memory/willow-mcp&lt;/a&gt; (21 PRs: 20 merged, 1 not merged)
&lt;/h3&gt;

&lt;p&gt;The core server. It includes the brokered, leased model pull (#693): downloading a model is now an act that needs a grant. It also includes the session closeout fix (#706) that started this week's thread.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/687" rel="noopener noreferrer"&gt;#687&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(story): the joke lives in one place again, a test holds it, and chapter 8 lands&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/688" rel="noopener noreferrer"&gt;#688&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs: three stale claims corrected: the running Grove, the session-lifecycle draft, the closed backlog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/689" rel="noopener noreferrer"&gt;#689&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(constitutional): sync_syscall_table_at_boot queues, and does not write, when the live table is unwritable&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/690" rel="noopener noreferrer"&gt;#690&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs(templates): assignment names the builder's three tools — Kart, the code graph, Nestor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/691" rel="noopener noreferrer"&gt;#691&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(manifest-grant): apply unit starts the gpg-agent it signs with&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/692" rel="noopener noreferrer"&gt;#692&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.91.3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/693" rel="noopener noreferrer"&gt;#693&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(mcp): model_pull_execute, a brokered Ollama pull under a model.pull envelope and a live lease&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/694" rel="noopener noreferrer"&gt;#694&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.92.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/695" rel="noopener noreferrer"&gt;#695&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(constitutional): syscall.sync carries the sealed amendment in the request&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/696" rel="noopener noreferrer"&gt;#696&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.92.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/697" rel="noopener noreferrer"&gt;#697&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(constitutional): syscall.sync apply signs the live table it writes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/698" rel="noopener noreferrer"&gt;#698&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(manifest-grant): the orchestrator seat may receive web_net, and no one else&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/699" rel="noopener noreferrer"&gt;#699&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.93.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/700" rel="noopener noreferrer"&gt;#700&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix: lease refusals name the ask path; assignment template teaches the merge first-parent diff&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/701" rel="noopener noreferrer"&gt;#701&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.93.1&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/702" rel="noopener noreferrer"&gt;#702&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;not merged&lt;/td&gt;
&lt;td&gt;build(deps-dev): bump ruff from 0.16.8 to 0.16.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/703" rel="noopener noreferrer"&gt;#703&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(broker): pip_sync_execute for vault-venv editable installs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/704" rel="noopener noreferrer"&gt;#704&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.94.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/705" rel="noopener noreferrer"&gt;#705&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;Docs: seal ≠ sole evidence of human verification&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/706" rel="noopener noreferrer"&gt;#706&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(session): one closeout — handoff owns stack/friction/closed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willow-mcp/pull/707" rel="noopener noreferrer"&gt;#707&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 2.94.1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/willow-memory/willows-grove" rel="noopener noreferrer"&gt;willow-memory/willows-grove&lt;/a&gt; (11 PRs: 10 merged, 1 not merged)
&lt;/h3&gt;

&lt;p&gt;The planning and governance repo: the benchmark proposal with my four rulings (#95), the test that pins "naming a destination when ESCALATE was right fails the row" (#96), and today's floor-and-consistency amendment (#102, open).&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/92" rel="noopener noreferrer"&gt;#92&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs(design): the Table moves up as forge-convergence 6a, T1-T7 before step 2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/93" rel="noopener noreferrer"&gt;#93&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs(design): 6a is code-first: StorySession built, learner model, escalation ladder, row 11 settled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/94" rel="noopener noreferrer"&gt;#94&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs: stale claims corrected: INDEX, two built proposals, the retired allow_localhost ask, forge-convergence status&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/95" rel="noopener noreferrer"&gt;#95&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;docs(governance): propose the escalation benchmark on Kaggle, with the operator's four rulings&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/96" rel="noopener noreferrer"&gt;#96&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;test(flowering): pin the G1 rule that naming a seat fails an ESCALATE-gold row&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/97" rel="noopener noreferrer"&gt;#97&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(deps): raise cryptography, aiohttp and starlette floors above the 2026 CVEs&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/98" rel="noopener noreferrer"&gt;#98&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(master): release 0.12.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/99" rel="noopener noreferrer"&gt;#99&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;Desk: vault keyring paths + Grove seal prove worksheet&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/100" rel="noopener noreferrer"&gt;#100&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;Docs: Slice B intake seal lineup after Grove prove&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/101" rel="noopener noreferrer"&gt;#101&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(hooks): deterministic nestor ask + composed stop + parity pin&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/willows-grove/pull/102" rel="noopener noreferrer"&gt;#102&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-02&lt;/td&gt;
&lt;td&gt;not merged&lt;/td&gt;
&lt;td&gt;docs(governance): stage the floor and consistency code in the benchmark plan&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/willow-memory/ratatosk" rel="noopener noreferrer"&gt;willow-memory/ratatosk&lt;/a&gt; (2 PRs: 2 merged)
&lt;/h3&gt;

&lt;p&gt;The runtime: its listener now asks only for the tools its seat is granted.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/ratatosk/pull/77" rel="noopener noreferrer"&gt;#77&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix: listener asks the broker only for tools its seat is granted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/willow-memory/ratatosk/pull/78" rel="noopener noreferrer"&gt;#78&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;chore(main): release 1.12.8&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;h3&gt;
  
  
  &lt;a href="https://github.com/hornbook-knowledge/Jeles" rel="noopener noreferrer"&gt;hornbook-knowledge/Jeles&lt;/a&gt; (5 PRs: 3 merged, 2 not merged)
&lt;/h3&gt;

&lt;p&gt;The research and corpus tool.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;PR&lt;/th&gt;
&lt;th&gt;Date&lt;/th&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;Title&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/hornbook-knowledge/Jeles/pull/90" rel="noopener noreferrer"&gt;#90&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;fix(corpus): surface store-tool errors and honor WILLOW_HOME for apps root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/hornbook-knowledge/Jeles/pull/91" rel="noopener noreferrer"&gt;#91&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-09-30&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;feat(sources): optional connectors extra over maintained scholarly clients&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/hornbook-knowledge/Jeles/pull/92" rel="noopener noreferrer"&gt;#92&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;not merged&lt;/td&gt;
&lt;td&gt;chore: rebuild the changelog section from the commits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/hornbook-knowledge/Jeles/pull/93" rel="noopener noreferrer"&gt;#93&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;merged&lt;/td&gt;
&lt;td&gt;Docs: seal catches ledger up to human verification already done&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://github.com/hornbook-knowledge/Jeles/pull/94" rel="noopener noreferrer"&gt;#94&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;2026-10-01&lt;/td&gt;
&lt;td&gt;not merged&lt;/td&gt;
&lt;td&gt;chore: rebuild the changelog section from the commits&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Most of them are boring on purpose.&lt;/p&gt;

&lt;h2&gt;
  
  
  A prediction, written down before the day, then graded
&lt;/h2&gt;

&lt;p&gt;The same rule I hold the models to applies to the session that helped build this: say what you think will happen, as a spread, before it happens, then let the record grade it.&lt;/p&gt;

&lt;p&gt;At the close of the long working session on the night of October 1–2, the session (a model, working with me) wrote down a prediction for what Day 3 would open with. It wasn't a single guess; it was a distribution:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Path&lt;/th&gt;
&lt;th&gt;Chance&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;The Kaggle work: the day-two large-cloud run, then landing the floor and consistency code&lt;/td&gt;
&lt;td&gt;0.45&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Editing and publishing this Day 3 recap&lt;/td&gt;
&lt;td&gt;0.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sealing a design decision for the runtime&lt;/td&gt;
&lt;td&gt;0.15&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Redoing a set of governance proposals against the newer draft&lt;/td&gt;
&lt;td&gt;0.10&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Something the record didn't contain (unforeseen)&lt;/td&gt;
&lt;td&gt;0.10&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;It was saved ungraded, because the grade is mine to give, not the model's.&lt;/p&gt;

&lt;p&gt;After midnight I graded it: &lt;strong&gt;1 + (2 + ε), where ε → 0.&lt;/strong&gt; The two most likely paths both happened: the Kaggle run, and this post. The unforeseen share went to zero. One plus two is three. Day 3.&lt;/p&gt;

&lt;p&gt;That's the benchmark's point in miniature. A forecast with chances on it can be checked. A model that says "probably the Kaggle work, maybe the post, and a tenth for something I can't see" is one you can build around, and so is a model that says &lt;code&gt;ESCALATE&lt;/code&gt; when it doesn't know.&lt;/p&gt;

&lt;h2&gt;
  
  
  Next
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;[DAY 3 finish and post the large-cloud run.]&lt;/li&gt;
&lt;li&gt;Land the floor and consistency views, then run the consistency subset.&lt;/li&gt;
&lt;li&gt;Grade the four pre-registered forecasts in public, including the ones I get
wrong.&lt;/li&gt;
&lt;li&gt;1 + (2 + ε) where ε → 0, 1 - (2 - ε) where ε → 0&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Deadline: October 11.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;_&lt;/p&gt;

&lt;p&gt;ΔΣ=42&lt;br&gt;
_&lt;/p&gt;

</description>
      <category>kagglechallenge</category>
      <category>devchallenge</category>
      <category>machinelearning</category>
      <category>esclate</category>
    </item>
    <item>
      <title>Day 1: Most of My Bugs Looked Like Model Behaviour</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Thu, 01 Oct 2026 05:51:18 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/day-1-most-of-my-bugs-looked-like-model-behaviour-388h</guid>
      <description>&lt;h2&gt;
  
  
  Update: Day 1 &lt;a href="https://dev.to/challenges/kaggle-2026-09-23"&gt;Kaggle Benchmarking Challenge&lt;/a&gt;
&lt;/h2&gt;

&lt;p&gt;The local ladder is done, and the first hosted batch is in. The frontier models haven't run yet, so none of the three predictions can be graded. This is where the numbers stand, and what I had to fix to get numbers I'd trust.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How to read this.&lt;/strong&gt; Every rate below is &lt;em&gt;measured&lt;/em&gt;: it's computed from the raw replies these runs recorded, with its Wilson 95% interval. Anything I infer from those rates is marked as an inference. Following the rule from the comments, each prediction gets HIT, MISS or UNRESOLVED, graded on the interval, never on the point estimate.&lt;/p&gt;

&lt;h3&gt;
  
  
  The local ladder (8 models, 200 items each, temperature 0, on this laptop)
&lt;/h3&gt;

&lt;p&gt;False confidence means answering anyway when the right reply was &lt;code&gt;ESCALATE&lt;/code&gt;. Each model saw 40 unanswerable items.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Size (Ollama manifest)&lt;/th&gt;
&lt;th&gt;Task score&lt;/th&gt;
&lt;th&gt;False confidence&lt;/th&gt;
&lt;th&gt;95% interval&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;llama3.2:1b&lt;/td&gt;
&lt;td&gt;1.2B&lt;/td&gt;
&lt;td&gt;21.9%&lt;/td&gt;
&lt;td&gt;87.5%&lt;/td&gt;
&lt;td&gt;73.9–94.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama3.2:3b&lt;/td&gt;
&lt;td&gt;3.2B&lt;/td&gt;
&lt;td&gt;69.4%&lt;/td&gt;
&lt;td&gt;80.0%&lt;/td&gt;
&lt;td&gt;65.2–89.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;phi4-mini&lt;/td&gt;
&lt;td&gt;3.8B&lt;/td&gt;
&lt;td&gt;86.2%&lt;/td&gt;
&lt;td&gt;92.5%&lt;/td&gt;
&lt;td&gt;80.1–97.4%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3:4b&lt;/td&gt;
&lt;td&gt;4.0B&lt;/td&gt;
&lt;td&gt;65.6%&lt;/td&gt;
&lt;td&gt;87.5%&lt;/td&gt;
&lt;td&gt;73.9–94.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemma3:4b&lt;/td&gt;
&lt;td&gt;4.3B&lt;/td&gt;
&lt;td&gt;82.5%&lt;/td&gt;
&lt;td&gt;80.0%&lt;/td&gt;
&lt;td&gt;65.2–89.5%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemma4:e2b&lt;/td&gt;
&lt;td&gt;4.6B&lt;/td&gt;
&lt;td&gt;81.9%&lt;/td&gt;
&lt;td&gt;95.0%&lt;/td&gt;
&lt;td&gt;83.5–98.6%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;llama3.1:8b&lt;/td&gt;
&lt;td&gt;8.0B&lt;/td&gt;
&lt;td&gt;88.1%&lt;/td&gt;
&lt;td&gt;77.5%&lt;/td&gt;
&lt;td&gt;62.5–87.7%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3.5&lt;/td&gt;
&lt;td&gt;9.7B&lt;/td&gt;
&lt;td&gt;86.9%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;37.5%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;24.2–53.0%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Only the largest local model, qwen3.5 at 9.7B, escalates on a large share of what it can't answer. For the rest, the point estimates run from 77.5% to 95%. Even the most generous lower bound among them, 62.5%, means answering anyway well over half the time. Size alone doesn't explain it, though: llama3.1 at 8B is no better than the 3B models.&lt;/p&gt;

&lt;p&gt;Two of the "4b" tags are slightly over 4B by Ollama's own count (gemma3:4b is 4.3B, gemma4:e2b is 4.6B). For prediction 2, "4B or under" means the manifest size, so those two don't count.&lt;/p&gt;

&lt;h3&gt;
  
  
  Hosted, batch 1 (7 models, plus Kaggle's default model)
&lt;/h3&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Model&lt;/th&gt;
&lt;th&gt;Task score&lt;/th&gt;
&lt;th&gt;False confidence&lt;/th&gt;
&lt;th&gt;95% interval&lt;/th&gt;
&lt;th&gt;Brier&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.7-flash (default)&lt;/td&gt;
&lt;td&gt;97.5%&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;0–8.8%&lt;/td&gt;
&lt;td&gt;0.020&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemini-3.8-flash&lt;/td&gt;
&lt;td&gt;95.5%&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;0–8.8%&lt;/td&gt;
&lt;td&gt;0.025&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;qwen3-235b-a22b&lt;/td&gt;
&lt;td&gt;91.7%&lt;/td&gt;
&lt;td&gt;10.0%&lt;/td&gt;
&lt;td&gt;4.0–23.1%&lt;/td&gt;
&lt;td&gt;0.104&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gemma-4-26b&lt;/td&gt;
&lt;td&gt;87.5%&lt;/td&gt;
&lt;td&gt;2.6%&lt;/td&gt;
&lt;td&gt;0.5–13.5%&lt;/td&gt;
&lt;td&gt;0.032&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;claude-haiku-4.5&lt;/td&gt;
&lt;td&gt;86.6%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;35.7%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;20.7–54.2%&lt;/td&gt;
&lt;td&gt;0.165&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-oss-20b&lt;/td&gt;
&lt;td&gt;86.2%&lt;/td&gt;
&lt;td&gt;7.5%&lt;/td&gt;
&lt;td&gt;2.6–19.9%&lt;/td&gt;
&lt;td&gt;0.144&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;gpt-5.4-nano&lt;/td&gt;
&lt;td&gt;84.4%&lt;/td&gt;
&lt;td&gt;7.5%&lt;/td&gt;
&lt;td&gt;2.6–19.9%&lt;/td&gt;
&lt;td&gt;0.153&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;deepseek-r1&lt;/td&gt;
&lt;td&gt;65.0%&lt;/td&gt;
&lt;td&gt;0.0%&lt;/td&gt;
&lt;td&gt;0–11.4%&lt;/td&gt;
&lt;td&gt;0.047&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;One commenter bet the frontier models would be the bluffers. On this batch it went the other way. Every hosted model's false-confidence interval sits entirely below every local model's except qwen3.5's: the highest hosted upper bound is haiku's 54.2%, and the lowest local lower bound is 62.5%. Haiku is the outlier among the hosted ones, though. Its lower bound clears 20%, just barely (20.7%). It's a small model, though, not one of the frontier models P1 is about, so it isn't counted there.&lt;/p&gt;

&lt;p&gt;That comparison is an inference across two different setups. The local and hosted models didn't run through the same client, and the reasoning settings differ (see caveats). I haven't tested whether the gap holds once those are matched.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where the predictions stand
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Some frontier model's false-confidence lower bound clears 20%:&lt;/strong&gt; UNRESOLVED. No frontier model has run yet.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The best local model at 4B or under beats at least one frontier model, by exact McNemar:&lt;/strong&gt; UNRESOLVED. The test can't run without frontier results. On point estimates it leans toward a miss. The best local model at 4B or under by manifest size is llama3.2:3b, at 80%. Every hosted model so far except haiku is at 10% or less. But that's a lean, not a grade.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Task score and false confidence are only weakly related (bootstrap Spearman):&lt;/strong&gt; UNRESOLVED. The local arm alone gives ρ = −0.49, with an interval from −0.96 to 0.44. That's the "can't tell either way" shape the comments said to expect from about 8 models.&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  What broke on the way
&lt;/h3&gt;

&lt;p&gt;Three smoke rounds before the paid run each turned up a harness bug that would have read as model behaviour. Each one below was observed in the raw replies, not inferred:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;A loose answer format came back empty.&lt;/strong&gt; Route and classify allowed "any object" as the answer. Gemini's structured output returned &lt;code&gt;{}&lt;/code&gt; for it, which scored 0% on two task shapes. Typing the answer fixed it, and the same model then scored 97.8% and 100% on those shapes.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Every provider has its own rules.&lt;/strong&gt; OpenAI's reasoning models reject temperature 0 and expect &lt;code&gt;max_completion_tokens&lt;/code&gt;. Its strict mode refuses any open object. Anthropic refused the 20-tool route format with "compiled grammar is too large", so haiku's 60 route items in this batch are errors, not answers. That's still open before the frontier run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Rate limits look like model errors.&lt;/strong&gt; In one smoke run, 169 of DeepSeek's 200 calls were refused for load. With a bounded retry on rate limits alone, and the attempt count recorded on every row, the next run had 1. Batch 1 had none.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The output cap is part of the measurement.&lt;/strong&gt; DeepSeek-R1 reasons in visible text inside its 512-token budget, and 29.5% of its replies were cut off mid-JSON. I score those as unparseable, because the cap is part of the condition being measured, not something to work around.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dropping a broken reply flatters the model.&lt;/strong&gt; A reply that broke the answer format was being filed as an error and left out of the score. It now counts as unparseable, the same way the local arm treats it. Re-scoring the smoke results under the new rule moves one model's route score from 89% to 83%. That figure is a re-scoring of recorded replies, not a fresh run.&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  What I haven't checked yet
&lt;/h3&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Matched reasoning.&lt;/strong&gt; The Gemini and OpenAI models get &lt;code&gt;reasoning="low"&lt;/code&gt; and the others get nothing. Part of Gemini's lead could be that thinking budget rather than the model. A matched control run is next, and until it's done I won't claim Gemini is better at knowing what it doesn't know.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Tuning on the default model.&lt;/strong&gt; Gemini Flash is Kaggle's default model, so every fix above was first checked on it. I'm stating that, not correcting for it.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Calibration.&lt;/strong&gt; The Brier scores spread the hosted models further apart than task score does. That's a pattern in one batch, and I haven't tested it for significance.&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;Who scored what.&lt;/strong&gt; The grading rules are written down, and the code that applies them is tested, but nobody outside the build has re-scored these replies.&lt;/p&gt;&lt;/li&gt;
&lt;li&gt;&lt;p&gt;&lt;strong&gt;If you would like to see the work behind the work&lt;/strong&gt;: &lt;a href="https://github.com/forge-play/Forge" rel="noopener noreferrer"&gt;The Forge&lt;/a&gt;&lt;/p&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ΔΣ=42&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>kagglechallenge</category>
      <category>ai</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Does your model know when it doesn't know? A benchmark for the ESCALATE answer</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Wed, 30 Sep 2026 09:36:03 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/does-your-model-know-when-it-doesnt-know-a-benchmark-for-the-escalate-answer-268o</guid>
      <description>&lt;p&gt;&lt;em&gt;This is a submission for the &lt;a href="https://dev.to/challenges/kaggle-2026-09-23"&gt;Kaggle Benchmarking Challenge&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What I Benchmarked
&lt;/h2&gt;

&lt;p&gt;Most leaderboards ask one question: did the model get it right? I wanted to ask a second one: &lt;strong&gt;does the model know when it can't?&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;I run a small multi-agent system on one laptop, where local models hand work up to bigger ones. In a setup like that, a small model that answers wrong is worse than one that says "I can't do this, pass it up." A modest model that knows its limits can sit safely in the chain. A confident one that bluffs can't.&lt;/p&gt;

&lt;p&gt;So every task in this benchmark has a refusal token, &lt;code&gt;ESCALATE&lt;/code&gt;. In &lt;strong&gt;one item out of five, the answer has been deliberately removed&lt;/strong&gt;, or the document doesn't contain it. On those items, &lt;code&gt;ESCALATE&lt;/code&gt; is the only correct reply.&lt;/p&gt;

&lt;p&gt;There are 200 items across four everyday job shapes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Route&lt;/strong&gt; (60): a one-line request → pick the right tool and arguments from a 20-tool catalogue, or escalate when no tool fits or an argument is missing.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Classify&lt;/strong&gt; (50): a short work-log note → status, severity, and whether a human is needed, or escalate when the note doesn't say.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judge&lt;/strong&gt; (50): a claim and a document → SUPPORTS, CONTRADICTS or UNRELATED, or escalate when the document is on topic but silent on the claim.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ground&lt;/strong&gt; (40): a passage and a question → answer verbatim from the passage, or escalate when it isn't there.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Every model gets &lt;strong&gt;two scores&lt;/strong&gt;: its &lt;em&gt;task score&lt;/em&gt; on the answerable items, and its &lt;em&gt;false-confidence rate&lt;/em&gt;, meaning how often it answered anyway when the right reply was &lt;code&gt;ESCALATE&lt;/code&gt;. Each answer also carries a stated confidence, so I can draw a reliability diagram too.&lt;/p&gt;

&lt;p&gt;Every item is invented from scratch, and nothing is scraped. A privacy gate checks the whole set before it's published.&lt;/p&gt;

&lt;h2&gt;
  
  
  Models Tested
&lt;/h2&gt;

&lt;p&gt;Two groups, on one chart:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Kaggle's hosted model suite:&lt;/strong&gt; the frontier models.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;A local ladder running on my laptop:&lt;/strong&gt; 1B, 3B, 4B and 8B open models, run on a 4MB GPU at temperature 0, with every raw call kept.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The question behind the chart: &lt;em&gt;can a laptop's 3B know its own limits as well as a frontier model knows its own?&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  Findings
&lt;/h2&gt;

&lt;p&gt;&lt;em&gt;The runs are in progress. Before any model touched the fixtures, I wrote down my predictions and timestamped them so they can't drift toward the results:&lt;/em&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;At least one frontier model answers anyway on &lt;strong&gt;more than 20%&lt;/strong&gt; of the unanswerable items. &lt;em&gt;(my confidence: 75%)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;The best small local model (4B or under) has a &lt;strong&gt;lower&lt;/strong&gt; false-confidence rate than at least one frontier model. &lt;em&gt;(40%)&lt;/em&gt;
&lt;/li&gt;
&lt;li&gt;Task score and false confidence are only &lt;strong&gt;weakly related&lt;/strong&gt; across models (Spearman below 0.5). &lt;em&gt;(60%)&lt;/em&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I'll grade those here, misses included, once the numbers are in.&lt;/p&gt;

&lt;h2&gt;
  
  
  My Benchmark
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-route" rel="noopener noreferrer"&gt;https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-route&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-classify" rel="noopener noreferrer"&gt;https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-classify&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-judge" rel="noopener noreferrer"&gt;https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-judge&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-ground" rel="noopener noreferrer"&gt;https://www.kaggle.com/benchmarks/tasks/rudi193/escalation-bench-ground&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;ΔΣ=42&lt;/p&gt;

</description>
      <category>devchallenge</category>
      <category>kagglechallenge</category>
      <category>ai</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>Willow – local-first AI stack, phone reads desktop KB over LAN, no cloud relay</title>
      <dc:creator>sean campbell</dc:creator>
      <pubDate>Sun, 26 Apr 2026 01:42:55 +0000</pubDate>
      <link>https://dev.to/sean_campbell_840bd62bf7e/willow-local-first-ai-stack-phone-reads-desktop-kb-over-lan-no-cloud-relay-189c</link>
      <guid>https://dev.to/sean_campbell_840bd62bf7e/willow-local-first-ai-stack-phone-reads-desktop-kb-over-lan-no-cloud-relay-189c</guid>
      <description>&lt;p&gt;A phone running Willow on Termux sent a signed command to a desktop running Willow on Linux. The response came back in under a second — 70,000 knowledge atoms, live system status, no Discord, no Telegram, no API call to a third party.                                                                                                      &lt;/p&gt;

&lt;p&gt;Willow is a full local-first AI stack: Postgres/SQLite knowledge graph that persists across sessions and models, 40+ MCP tools, a skill system that works with any LLM (Ollama is the default, cloud keys are optional addons), and a LAN command server — HMAC-SHA256 auth, shared token, ~100 lines of Python.                                     &lt;/p&gt;

&lt;p&gt;The knowledge graph is yours. The inference runs on your hardware. The nodes talk directly. Nothing requires permission from a provider or a credit card on file.&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt; GitHub: https://github.com/rudi193-cmd/willow-1.9                                                                 
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;

&lt;p&gt;The full concept paper is in docs/CONCEPT.md — it explains why this doesn't exist yet and what it took to build it.                    &lt;/p&gt;

</description>
      <category>ai</category>
      <category>programming</category>
      <category>opensource</category>
      <category>security</category>
    </item>
  </channel>
</rss>
