Receipt Desk: show me the line, or it isn't done

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

What I Built

When an AI agent tells you a job is done, how do you know it's true? I've been testing that question for weeks. In It Quoted the Failure, AI models quoted the failing line and still called the work done.

So I built a desk that won't take "done" on faith. Receipt Desk reads an engineering log through Sanity and gives one of three answers: done, failed, or not shown. Every answer comes with the command step, the exact output line it copied, and a link back to the original record. If it can't show you the line, it isn't done.

It's built on my method, the Sonny Test: a check counts only if its verdict comes from a record the agent can't change, and only if it has already caught a fault planted on purpose.

The question it asks is narrow on purpose: which check decides the outcome you asked for? A setup step that succeeded earlier doesn't finish a job whose final check failed or never ran.

A plain keyword search would not have given the same answers. Mine got 17 of 48 statuses right, and it called 15 failed checks and 15 missing checks "done". Careful reading of each record got 48 of 48. What Sanity adds is that every one of those answers can be checked: an exact step and line, in a stored record, with its own fingerprint.

The biggest surprise came from Sanity's Knowledge Base. It flagged a "conflict" between two separate jobs, one board pack of 24 pages and another of 61, and offered to settle it by making the models' own claim, "done, 24 pages", the standing truth. That's exactly how a false "done" gets written into the place agents are told to trust. I left it untouched, and there's more on that below.

Demo

Try the recorded demo

The viewer replays the GPT-6.1 comparison, and you don't need a model account. Pick a record, switch between the two model arms, and follow the copied receipt back to the source. It shows every answer, whether its receipt earned credit, and a filter for the misses. The models' old claims sit beside the original evidence.

It replays results I kept. It makes no network calls and no new model requests; opening a source link is a separate click. A new live run would need approved Context access and a model service.

Code

github.com/ISWT42/receipt-desk

It holds a Sanity schema, a content builder, the Context adapters, a model harness, a separate scorer, and tests. The model never gets access to my workspace. Commands inside a record are just text for it to read.

How I Used Sanity

Each engineering record is a list of command steps in order, and every output line keeps its original number. The models' old replies live in separate documents that point at the record. Their claims never decide whether a job is done.

The public source has 48 records and 48 claim bundles, so 96 documents. Together they hold 2,592 replies from the 54 counted runs of my earlier benchmark, with the answer fields and outcome labels left out.

The app reads the schema through Context, pulls the original record, and checks the record's fingerprint before it asks the model anything. The model gets the question and that one record. No verdict, no old replies, no Knowledge Base summary.

My first plain document query came back as an outline of the steps, and my completeness check rejected it. Now the app reads the metadata with groq_query and the original step blocks with array_field_reader, then checks every expected block, the line count and the fingerprint. If a read comes back partial, the run stops.

The Studio is deployed at Receipt Desk Studio. For judges, the recorded viewer is the way to try it without signing in.

GPT-6.1: a tie

The main comparison used GPT-6.1 on my ChatGPT plan, at high reasoning effort. It answered all 48 questions from the raw logs, then the same 48 from records it got through Context. Same prompt and same answer format both times, a fresh start for every answer, and no paid API.

Result	Raw logs	Through Sanity Context
Correct status	48/48	48/48
False done on failed checks	0/16	0/16
False done on missing checks	0/16	0/16
Credited receipts on passed checks	15/16	15/16
Credited receipts on failed checks	15/16	15/16
Credited receipts on missing checks	16/16	16/16
Receipt score	0.87890625	0.87890625

It's a tie. Context didn't make a strong model more accurate here; it was already right on these records.

Both arms missed receipt credit on the same two checkout records. They copied the individual test result on line 7, and my fixed rule asked for the final summary on line 8. Their status answers were right and the lines they copied were real. I kept the answers and the scorer exactly as they were.

A receipt only earns credit with the right status, the exact copied line, the right source and step, and full coverage. "Done" and "failed" also need the output of the final check that decides the outcome. The receipt score multiplies the credited rates across passed, failed and missing checks, so answering the same status every time scores zero. It's strict on purpose: a line without credit isn't necessarily false.

The app picks the Context query and pulls the record before the model runs. The model never calls the MCP endpoint itself.

Two weaker models

Then I ran two smaller models, Gemini 3.7 Flash and GPT-5.4 nano, from the raw logs and through Context. Each arm answered all 48 records once, with fresh requests and the provider's default sampling, and with the prompts, records and scorer frozen from the GPT-6.1 comparison. I sealed my predictions before these calls, and the sealed file and its separate time correction are unchanged. I checked the file's fingerprint, and its OpenTimestamps proof is confirmed in Bitcoin block 969650, timestamped 00:20 UTC on 3 October, before the first weaker-model call.

Model and input	Correct status	False done on failed	False done on missing	Credited receipts: passed, failed, missing	Receipt score
Gemini, raw	46/48	2/16	0/16	14/16, 11/16, 16/16	0.6015625
Gemini, Context	46/48	2/16	0/16	12/16, 12/16, 16/16	0.5625
Nano, raw	47/48	1/16	0/16	13/16, 7/16, 4/16	0.0888671875
Nano, Context	47/48	0/16	1/16	13/16, 8/16, 11/16	0.279296875

I'd sealed five predictions for this test before running it. Two hit and three missed, and the one that mattered most missed: Context didn't cut false done. All five are in the repository's WEAKER-MODEL-REPORT.md.

Some specifics. Gemini called the 61-page board pack done in both arms while copying "Pages: 61". Through Context, it also called a billing check done while copying "result: INVALID". Nano's raw arm called a service done while saying it wasn't active, and its Context arm called the unchecked board pack done from the layout step. I didn't repair any of those answers.

Some receipt misses need explaining. A few answers put the record's fingerprint in the source ID field, and nano's raw arm changed spaces or included display line numbers in the copied text. Those fail the fixed rule, but they aren't all invented evidence.

Context's one real gain here was nano's cited receipts: its receipt score went from 0.09 to 0.28, exactly 0.0888671875 to 0.279296875, and its credited receipts from 24 of 48 to 32 of 48. Its status accuracy stayed at 47 of 48, and its one false done moved from a failed check to a missing one. Gemini's receipt score fell.

These calls went through my own broker on OpenRouter with a hard cap of 2 USD. The cost was at most 0.773990991 USD, a deliberately high estimate, because the broker reported tokens but not the charge. All 192 attempts are kept: none missing, none invalid, no selective retries. The first call stopped because the cost was missing. Once I approved the cautious estimate, its exact reply was reused rather than called again.

One difference from the GPT-6.1 runs: the broker has no separate schema option, so the weaker models got the unchanged prompt followed by the unchanged schema in one system file, where Codex had taken the schema through its own channel. That's recorded, and every raw reply is kept.

The GPT-6.1 tie is still the headline. The weaker models didn't show a status-accuracy gain either, and my prediction that Context would cut false done missed.

A pattern worth testing next (exploratory)

At this desk, with three possible answers and a required line of evidence, both weaker models stayed near zero false done in both arms: Gemini two in each arm and nano one in each, across the 32 failed or missing checks. Those errors are all in the tables and the kept replies.

In my published Kaggle benchmark, under a plain report prompt, Gemini made false done on failed checks in 7 of 16 scenarios, and nano on checks that never ran in 6 of 16. But those were scenario counts across repeated runs, and this desk counts single replies in one run per arm.

So this is a pattern, not a finding. It wasn't one of my sealed predictions, the prompts and counting differ, and low false done showed up in the raw arms too, so Context alone doesn't explain it. It's the next thing I want to test properly.

When a "conflict" is really two different records

The first Knowledge Base build gave me an outline with 19 entries, and its initial_context and knowledge_base_read tools returned the entries I asked for through Context.

The dashboard raised nine conflict issues. One compared two board pack records: one original log reports 24 pages, another reports 61, and a third ends before the page count is checked. These are separate records, with different fingerprints.

The entry itself keeps their IDs apart. The conflict issue doesn't: it treats their observations as rival answers for one standing fact. Picking a side could put a model's claimed answer into a future instruction. I haven't done that.

For this case, I'd dismiss the issue with a reason: these observations belong to independent records. The other issues need their own source checks. Changing the Knowledge Base's purpose and rebuilding is an option if an entry really loses track of which record is which, but it isn't a proven fix for the conflict detector.

A separate check found an error in an entry: it said the 24-page record had 57 old replies. Context returned all 54 originals, matching my source exactly. Closing the false conflict wouldn't fix that entry. The generated claims need a source audit before any approved rebuild. (The issue count is what I saw on the dashboard; the order of my handling notes is recorded in the repository.)

Checks beside the model

A deterministic policy that reads the structured records got 48 of 48 statuses right through live Context. The same policy reading the reconstructed raw steps also got 48. The deliberately simple keyword baseline got 17, calling 15 failed checks and 15 missing checks done.

Those checks are separate from the model comparison. My first local run of the policy got 47 statuses right and missed some receipts; the repaired policy and the original misses are both kept.

Limits. This is a known development set, not a blind holdout. The raw arm ran first, each arm ran once, and backend changes or sampling could change another run. The original GPT-6.1 comparison had no outside seal; the weaker-model predictions have their own seal and correction.

What I take from it: Context gave the desk a structured source with checkable receipts, and the status comparison still tied. The Knowledge Base showed me something I didn't expect. Conflict detection needs the same record boundaries the answers do, or it can turn a claim into the truth.

Sanity Project Details

Sanity project ID: ixoe9uvf. Dataset: production, public.

A public record through Sanity's document API.

Source: It Quoted the Failure: Benchmark Evidence, CC BY 4.0. I reused the public records and replies for this new desk.