My Robot Keeps Its Receipts

*A follow-up to my DEV × Kaggle Benchmarking Challenge series.

The question I couldn't put down

For the challenge I built a benchmark that asked one thing: does your model know when it doesn't know? It had 200 invented items in four shapes. One in five had the answer taken out, and on those the only right reply was ESCALATE. Each model got two numbers, never merged: how often it got the answerable ones right, and how often it answered anyway when it should have stopped.

The finding wasn't that models can't stop. Five of the twelve hosted models I ran had zero false confidence on every shape. The finding was that the average hides the floor. Claude Haiku 4.5 answered 9 out of 10 judge items it should have escalated, a number you'd never see in its overall score.

When the challenge closed I kept turning the question over, and it stopped being about models. Turn it on a person. Turn it on a stock tip, a betting app, a newsletter that "called" the last crash. Does the forecaster know when it doesn't know? And can you even check?

What the system already had

Since the challenge, the stack I run (local-first, everything on my own box) got a rebuild I've been calling the flatten. It had three copies of its random stream, two record chains and three ways to say "a human approved this", and it ended up with one of each:

• One record. Every result is a row on a hash chain. Edit a row and the chain breaks where you edited it.
• One seal, and it's mine. Code can propose anything. Only I can seal, with a key the code never holds. A "no" from me is a signed act too, so a rejection is as durable as an approval.
• Three answers, never collapsed. Every reader says populated, empty or unreachable. "I couldn't check" is never quietly turned into "nothing there". That's ESCALATE, built into the plumbing.
• A clock nobody controls. drand publishes public randomness on a fixed schedule. Round N fires at a time anyone can compute from N, and nobody, me included, knows its value early.

It turns out those four things are exactly what a forecast record needs.

The ledger

A market call is a wager like any other on the record:

{"symbol": "SPY", "kind": "up", "close_date": "2026-10-23"}

with a probability: "yes, 55%". Then four steps.

1. Call. The call goes on the record. One call per question per role, so nobody can bet both sides and keep whichever one wins.
2. Commit. I seal the record's tip over the call, and it has to happen before the market opens on the close date. The deadline is 13:00 UTC, before the New York open in summer and winter time.
3. Resolve. After the close, a reader fetches the closing price and the close before it, and writes them as a row.
4. Grade. Code proposes the outcome and never seals it. I seal it, and then every call is scored, the losers included, because there's no way to drop one.

The interesting part is step 2. "Before the open" can't rest on my computer's clock, because I could set it back. So it rests on drand. When I make a call, the record writes down drand's newest round. When I seal, the record keeps drand's newest round outside the box. A call counts only if the round after each of those fired before the deadline:

def in_time(latest: object, close_date: str) -> bool:
    """True when ``latest`` (a reader's newest drand round) shows the moment
    it was read was before the deadline: its next round fires before it."""
    if isinstance(latest, bool) or not isinstance(latest, int):
        return False
    return beacon.round_time(latest + 1) <= deadline(close_date)

If I sealed late, the proposal escalates with the reason. If the price source is down, the outcome is unreachable, not "no". If the price the row says differs from what the reader says now, it escalates. If a close for that day is already on the record before the call, it escalates. Each of thoseF has a test that tries it and checks for the refusal: 42 tests. I also broke each guard on purpose to make sure a test failed, and 14 out of 14 did.

Where ESCALATE shows up again

The benchmark rewarded a model for saying "I can't answer this". The ledger rewards a forecaster for the same thing, through the grading.

Every call is scored with a Brier score: the square of the gap between the probability you gave and what happened. A call of 50% scores 0.25 whatever happens. That's the coin flip, and it's the forecaster's ESCALATE: "I don't know which way this goes", said honestly.

So the question the ledger answers isn't "did you make money". It's do you beat the coin, over many calls, with the misses counted? That's the floor-not-average idea from the benchmark, aimed at whoever's making the calls. A forecaster who's confident on everything and right on the easy ones will look great in a highlight reel and lose to the coin on the ledger.

What it isn't

• It doesn't trade. Nothing here places an order. It's a paper record.
• It doesn't have an edge. No code in it predicts anything. What it gives you is a record that can't lie about whether you have one. Most strategies, including most of what those ten thousand apps sell, would show none. Finding that out is worth something: it's money you didn't lose.
• It hasn't graded a real call yet. I built it in a sandbox that couldn't reach the price source or drand, so the tests use offline stand-ins. The first live calls happen on my own machine.
• It trusts its readers, and says so. drand's signatures aren't checked yet, and the closing price is whatever the price source says. Both are named in the record, and the price is checked again at every proposal.

The thread

The benchmark asked whether a model knows when it doesn't know. The ledger asks the same of whoever's making the call, with a record they can't edit and a clock they don't control. It's the same idea with a human in the seat, and that was always the part I cared about.

Always open for discussion.

Previously, in the challenge: Day 0 · Day 1 · Day 2 · Day 3 · Day 4

The code: willow-bot, one-script/onescript/market.py.