DEV Community: Kai Chen

Shoebox: a small model reads your receipts overnight, slowly, and never guesses a number

Kai Chen — Tue, 20 Oct 2026 17:15:40 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Receipts are the paperwork nobody does on time. They go into a wallet, then a drawer, then a shoebox, and at the end of the month (or the end of the tax year) someone sits down and types them into a spreadsheet one by one: date, shop, total, category, then files the paper or the photo somewhere it can be found again. Every freelancer and small-business owner knows the evening it takes.

Shoebox is a folder that does that evening for you while you sleep. You photograph receipts whenever you get them and let the photos land in Shoebox/Inbox (phone photo sync, a shared drive, anything). Every night at two, Shoebox wakes up and works through the new ones, one at a time, on your own computer:

Gemma 4 looks at each photo and copies down what is printed: every line, the subtotal, discount, tax, total, cash paid and change.
The arithmetic decides whether to believe it. A receipt is filed only when its total is confirmed by other figures printed on the same receipt.
Each believed receipt is renamed (2026-10-14 Bakery Ana 60000.jpg), filed under its month and category, and added to the month's expense sheet.
A receipt whose figures will not agree is never guessed at. It goes to Check me/ with a note saying exactly what disagreed.

In the morning there is one file waiting, Good morning.txt:

Filed 7 receipts overnight; 3 would not add up and wait in “Check me”.
This month so far: 756,127 (Books/expenses-2026-10.csv).

And if you want to know something, you ask in plain words:

$ shoebox ask "How much did I spend on food and drink this month, and what is waiting for me to check?"
In October 2026, you spent 724,627 on Food & drink. Three receipts are waiting for you to check, with reasons provided for each one.

It is slow on purpose. It runs a 4-billion-parameter model on the CPU, so it can share a laptop with everything else you do, and because it has all night it can afford to look twice. That is the trade this week's theme asked for: nothing about it is fast, and you never have to think about it.

Demo

Ten receipt photos from a month, dropped into the inbox (they are from the CORD dataset, so they are Indonesian and in rupiah). After one night:

Shoebox/
  Inbox/  (empty)
  Filed/2026-10/Food & drink/
    2026-10-04 ตั้ง 24000.png
    2026-10-06 Unknown shop 148127.png
    2026-10-07 Unknown shop 93500.png
    2026-10-09 Unknown shop 54000.png
    2026-10-10 Unknown shop 40000.png
    2026-10-17 Unknown shop 365000.png
  Filed/2026-10/Other/
    2026-10-01 Unknown shop 31500.png
  Check me/
    2026-10-03 Unknown shop 113886.png  (+ .txt)
    2026-10-13 Unknown shop 60394.png   (+ .txt)
    2026-10-17 Snack Lestari All 48000.png  (+ .txt)
  Books/
    expenses-2026-10.csv, summary-2026-10.html, ledger.jsonl
  Good morning.txt

(Most CORD receipts have the shop's name blanked out for privacy, hence "Unknown shop"; one name came back in Thai from a logo.) Every file also gets a .json beside it with everything the model copied and every check that was made.

The month's summary, written for a person to read or print:

And this is what a receipt that will not add up looks like in the morning. The real total of this one is 68,000; the model read 48,000 the first time:

The figures on this receipt did not agree, so it was not filed:
- the two readings of the total disagree (48000 and 100000)
- paid 100000 minus change 32000 is not the total 48000

Retake the photo (flat, the whole receipt, the totals sharp), or file it by hand.

Code

luoy16002-svg / shoebox

A shoebox for receipts that empties itself overnight: Gemma 4 reads them on your own computer, Temporal keeps the night shift going, Mastra keeps the books.

Shoebox

A shoebox for receipts that empties itself overnight.

Photograph receipts whenever you get them and let the photos land in one folder (your phone's photo sync, a shared drive, anything). Every night Shoebox reads the new ones on your own computer with Gemma 4, checks each one's arithmetic, files it by month and category with a sensible name, and writes the month's expense sheet. In the morning there is a two-line note:

Filed 7 receipts overnight; 3 would not add up and wait in “Check me”
This month so far: 756,127 (Books/expenses-2026-10.csv).

Nothing is uploaded anywhere. The model runs locally, the night shift survives the laptop sleeping or the process being killed halfway (it is a Temporal workflow), and you can ask about the books in plain words:

$ shoebox ask "How much did I spend on food and drink this month, and what is waiting for me to

…

View on GitHub

How I Built It

Three open pieces, each doing one job.

Gemma 4 copies; it never calculates

The reader is Gemma 4 E4B (Apache 2.0) in llama.cpp, quantized to Q4_0 and running on six CPU threads. The prompt asks it to copy every money line exactly as printed, separators and all ("60.000", "Rp. 91,000", "12.50"), and the request carries a JSON schema that llama.cpp turns into a grammar, so the answer always parses.

Copying instead of calculating matters more than it sounds. Receipts from the test set I used print "60.000" for sixty thousand rupiah, "26,818" for twenty-six thousand, and "Rp. 91,000" with a dot that belongs to the currency. If the model normalised those numbers it would make a different mistake on each format. Instead the code reads the strings: a separator followed by exactly three digits groups thousands, a single one followed by one or two digits marks cents, and anything stuck to the currency sign is ignored. That parser has unit tests; the model never has to know any of it.

The first version had thinking turned on (Gemma 4 thinks by default in llama.cpp's chat template). It wrote a long "Here's a thinking process..." before every receipt, took about 75 seconds each, and on one receipt spent its whole token budget thinking and returned nothing at all. Copying a receipt needs eyes, not deliberation: with --reasoning off one look takes a median of 15 seconds, five times faster, and the numbers below are all measured that way.

The arithmetic decides, and the receipt is read twice

Every receipt prints its own proof. The lines add up to the subtotal. The subtotal, with discount, tax (included in the prices or added on top) and service, makes the total. The cash handed over minus the change is the total. Shoebox checks every one of these it can make.

Getting from there to a rule I trust took three tries, and I measured each one on the 95 receipts in the CORD v2 test split (real Indonesian receipts, CC BY 4.0) that print a total.

Try one: every figure must agree. On the first night it set aside four of five receipts, and all four had the right total. The model had copied the TOTAL and CHANGE lines into the item list, or written a ticket's unit price where the line meant quantity times price, so the lines never added up even though the total was fine. Too strict to be useful.

Try two: the total must be confirmed by something else on the receipt. Cash minus change, or the lines, or the subtotal. Lines that are obviously totals or payments ("TL", "CG", "TUNAI", "KEMBALI") are taken out before adding, and a line can count as quantity times price. This filed 87 of the 95 receipts, and 4 of those had the wrong total. Those were the interesting ones, because the arithmetic agreed with itself:

the model missed some lines, then wrote a subtotal and a total that matched the lines it had seen;
a shop printed "Rp 165000.00" and the model copied every amount on the receipt without the decimal point, so the lines, the subtotal, the total, the cash and the change were all ten times too big and all agreed;
the total and the change were misread by the same 5,000, so cash minus change still came out right.

A model can be consistently wrong, and arithmetic on its own output cannot see it.

Try three: read it twice. Now every receipt gets two looks. The first reads the whole receipt; the second sees only the bottom of the photo, cropped and enlarged, and copies the totals block. Like double entry, the two readings must give the same total. When they do, they are merged (a tax line the first look missed comes from the second) and the arithmetic decides. When they do not, a reading is used only if cash and change confirm it, because those are two figures printed apart from the total; otherwise the receipt goes to Check me/.

	one look	two looks
Total read correctly	88 of 95	88 of 95
Filed	87	80
Filed with a wrong total	4	1
Set aside in `Check me/`	8	15
... of which the total was in fact right	5	9

The full table, with every receipt that was not filed and why, is in eval/RESULTS.md, and npx tsx eval/cord.ts reproduces it.

The row that matters is the bold one. A receipt filed with a wrong total is a silent error in your books; a receipt in Check me/ costs you ten seconds in the morning. Reading twice cut the silent errors from four to one, at the price of seven more receipts to glance at. The one that still slips through is a receipt that prints its total before rounding (680,339) and after (680,500); both looks copied the first, and the arithmetic agrees with it. The two looks together take a median of 20 seconds per receipt on six CPU threads.

Temporal keeps the night shift going

A night shift on a laptop gets interrupted: the lid closes, Windows restarts for an update, the process dies. The shift is a Temporal workflow so none of that matters.

A schedule starts the nightShift workflow every night at 02:00 (shoebox start --at 02:00).
nightShift lists the inbox and starts one child workflow per receipt, one after another, so the CPU is never asked for two at once.
Reading a receipt is an activity that sends a heartbeat every ten seconds while Gemma works. The heartbeat timeout is one minute, so if the worker dies mid-receipt, Temporal notices within a minute and retries the read with backoff when the worker comes back.
Filing checks the ledger before it moves anything, so a retry after a crash never files a receipt twice.

I tested it the rude way: started a shift, killed the worker in the middle of a receipt, waited, and started it again. At 00:23:22 UTC the fourth receipt's read started. At 00:23:30 I killed the worker. For a minute and a half nothing happened: no worker, the shift's state sitting in the Temporal server. At 00:25:13 I started the worker again. Temporal had already marked the read as failed by heartbeat timeout, scheduled it again, and the new worker picked it up. The receipt's event history (temporal workflow show -o json), trimmed to the fields that matter:

event 5  ActivityTaskScheduled  00:23:22  readReceiptActivity
event 6  ActivityTaskStarted    00:25:16  attempt 2
         last failure: activity Heartbeat timeout
event 7  ActivityTaskCompleted  00:25:45

The shift carried on and finished the other six receipts. The ledger has every receipt exactly once. Here is the shift in the Temporal UI; the long bar in the middle is the receipt that was being read when the worker died:

The whole Temporal server is the open-source dev server (temporal server start-dev) with its history in a SQLite file in your home folder. Nothing about it needs a cloud account.

Mastra keeps the books

Two small agents run on the same local Gemma through Mastra, pointed at llama.cpp's OpenAI-compatible endpoint:

The filing clerk puts each believed receipt in one of eight categories, using Mastra's structured output with an enum, so it cannot invent a ninth.
The bookkeeper answers questions about the ledger. It has three tools (list expenses by month, category or shop; total a month by category or shop; list what is waiting in Check me/) and instructions to take every number from a tool. The model chooses which tool to call and phrases the answer; the amounts come from the books.

Why Does Open Innovation Matter?

A shoebox of receipts is a diary. It says where you were, at what time, what you ate, which pharmacy you went to, and often the last four digits of your card. Most receipt apps that do this job today work by uploading the photos to their servers and charging a monthly fee per user. Shoebox does the same job with the photos never leaving the folder they were taken into.

That is only possible because the model's weights are open. Gemma 4 E4B is small enough to run on a CPU, licensed so anyone can, and good enough at reading print that the arithmetic around it can catch the rest. Llama.cpp makes it a local server in one command; Temporal and Mastra are open source and run on the same machine. There is no API key anywhere in the project, no bill that grows with the number of receipts, and nothing to stop working when a vendor changes its pricing.

Open weights also made the design possible. Because the model is mine to run, I could turn its thinking off, constrain its output with a grammar, give it all night, and ask it to look twice. None of those are knobs a per-request API makes cheap.

The Lazy Payoff

What it takes from you: photos dropped in a folder whenever you get a receipt, and a look at Check me/ in the morning. What it takes from the computer: about 20 seconds of CPU per receipt at two in the morning, so a month of forty receipts is a quarter of an hour while you sleep.

What it saves is the evening at the end of the month: typing each receipt's date, shop, total and category into a spreadsheet, renaming the photo and putting it somewhere you will find it again. On this test set, five out of six receipts never need you at all, and the sixth comes with a note that says exactly which figures to look at.

Prize Categories

Best Use of Gemma: Gemma 4 E4B reads every receipt and powers both agents, locally on the CPU through llama.cpp.
Best Use of Temporal: the night shift is a scheduled Temporal workflow with a child workflow per receipt, heartbeating activities and idempotent filing; killed mid-shift, it picks up where it stopped.
Best Use of Mastra: the filing clerk (structured output) and the bookkeeper (tools over the ledger) are Mastra agents on the local model.

Tickmark: bank statements to spreadsheets, checked to the cent, without uploading them

Kai Chen — Thu, 15 Oct 2026 12:48:08 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Every month, somewhere, a bookkeeper is retyping a bank statement. The bank hands over a PDF; the books want rows in a spreadsheet, or an import file for QuickBooks or Xero. People pay to skip that job. A one-person web app that does nothing but convert bank statements was reported this year at about $38,000 in monthly recurring revenue, and freelance boards carry a steady stream of "PDF statement to Excel" requests.

Two things keep bothering the people who do this work.

The first is that the converters want the statement uploaded. For an accountant, that means sending a client's bank history to a server they know nothing about.

The second is trust. A converter that reads a 1 as a 7 does not tell you. You find out at month end, when the books are six dollars off and you go hunting through two hundred rows.

Tickmark is a web page that turns a bank statement PDF into a spreadsheet on your own computer, and checks every row while it does it. Each amount is added to the running balance, and a row gets a tick only when the balance the bank printed beside it agrees to the cent. A row that does not agree is ringed in red, with the size of the gap and the place on the page to look.

Scanned statements, which have no text to copy, are read by Gemma 4 running on your own graphics card through WebGPU. The model writes the rows down; the same arithmetic then decides which of them are right.

Who pays for this: bookkeepers, accountants and small business owners who convert statements every month, the same people already paying the upload-based converters. The plan is a free open core and paid extras that save hours: a year of statements in one go, rules remembered per client, column mappings for each accounting package.

Demo

Try it: https://luoy16002-svg.github.io/tickmark/

It opens with seven made-up statements from invented banks, printed the different ways real banks print them: one table with a signed amount column; paid in and paid out columns with an overdraft and one balance a day; decimal commas and two date columns; CR and DR markers; sections by kind (deposits, withdrawals, checks, fees) with no running balance; a check-number column with credits, debits and an ending daily balance; and sections with unsigned amounts and a separate daily balance table. There is a scanned one too. Or drop your own PDF on the page. The counter at the bottom shows every request the page makes to another site after you open a file. It stays at 0.

Here I changed one figure by hand, 27.32 to 37.32, to show what a misread looks like. The ledger rings the printed balance in red, says the gap is exactly 10.00, marks the line on the bank's page, and picks up again on the next row:

A scanned statement being read by Gemma 4 in the browser. The frame on the page is the band the model is reading; the rows appear as it writes them:

It works on a phone too, minus the scanning:

Exports: an Excel workbook (a transactions sheet and a reconciliation sheet), CSV, QuickBooks Online CSV, Xero CSV and OFX.

Code

luoy16002-svg / tickmark

Bank statements into spreadsheets, checked row by row against the running balance, in your browser. Scans are read by Gemma 4 on your own GPU.

Tickmark

Bank statements into spreadsheets, checked row by row, on your own computer.

Open a PDF statement in the browser. Tickmark copies every transaction into a ledger and adds it to the running balance. A row gets a tick only when the balance the bank printed agrees to the cent; a row that does not agree is ringed in red, with the gap and the place to look. Scanned statements are read by Gemma 4, Google's open model running on your graphics card through WebGPU, and go through the same check, so a figure the model misreads shows up as a row that does not tie instead of slipping into your books.

Nothing is uploaded. The page counts every request it makes to another site and shows the number at the bottom.

What it reads

Text PDFs as banks issue them: one table or sections by kind (deposits, withdrawals…

View on GitHub

How I Built It

Reading the text layer. pdf.js gives every piece of text on a page with its position. Tickmark groups pieces on the same baseline into lines, joins pieces closer than about half a character into cells, finds the header line (Date ... Balance, in a dozen spellings), and assigns every figure to the column whose right edge it lines up with. Dates come as 09/01, 01 Sep 26, SEP 01 or 01.09.2026, so the statement period is read first and each date takes its year from it. Amounts come signed, in separate in and out columns, with CR and DR, in brackets, or with decimal commas, and an overdrawn balance can be printed three different ways. Some banks print one long table; others print a section per kind of transaction with the amounts unsigned and the balances in a separate daily table. Tickmark takes the direction of the money from the section title, lists such rows by date and checks them day by day. When a PDF runs two headings together into one piece of text ("Deposits/Credits Withdrawals/Debits"), it splits them again at the phrases banks use.

The check. The running balance is the bank's own arithmetic, printed on the statement. Tickmark walks the rows, adds each amount, and compares. When a statement prints a balance only at the end of each day, the day's rows are checked together. When an amount is printed without a sign, the balance settles it: if only one choice of in and out makes the day add up, that is the one used. When nothing adds up, the row is flagged and the running balance picks up again from the next printed figure, so one bad row does not turn the rest of the page red.

Gemma 4 in the browser. Transformers.js 4.3 runs Gemma 4 E2B (q4f16) on WebGPU, in a web worker so the page stays responsive. The first time, the browser downloads 3.38 GB of weights from Hugging Face; after that it loads from the browser's cache in about five seconds on my machine (an RTX 5060 Ti).

My first attempt gave it a whole page, and the result was useless: three rows right, then shifted amounts, repeated rows and missed lines. The reason was in the image processor. The web build of the vision encoder turns any picture into 280 tokens, and at that budget a page of 9 point print is a blur. Raising the budget failed twice (first a WebGPU buffer over 2 GB, then an encoder fixed at 280 features), so I went the other way: the page is cut into ten overlapping bands and the model reads one band at a time. At that scale the print is legible.

What comes back still needs care, and every repair is one the arithmetic can prove:

a row read twice where two bands overlap is dropped, because a real transaction cannot leave the balance unchanged;
the second printed line of a description, read as a row of its own carrying the line above's balance, is joined back to its row;
a row that the next row ties without is not on the statement;
blank paper is never shown to the model, because shown an empty band it sometimes invents an example table.

On the scanned sample (three pages, 64 transactions) Gemma 4 reads every row correctly in about two minutes, and all 64 tie. On the UK sample, which prints one balance a day and leaves many cells empty, it does much worse: in my benchmark it got 23 of 58 rows exactly right. The check still does its job there. Every day that does not add up is marked, in the ledger and in the export, so the errors are in plain sight, but you would be correcting a lot by hand. Reading one row at a time, with the column headings pasted above it, is what I am working on next. The benchmark page in the repository runs the scan pipeline on any sample and scores it against the sample's answer key.

Categories. Plain rules handle the common payees: payment processors, utilities, couriers, fuel, taxes, bank fees. Gemma 4 suggests the rest, one question per payee, and any answer that contradicts the direction of the money (an expense category on money that came in) is thrown away. Suggestions stay in italics until you confirm them, and your choices are remembered for next month.

Proving nothing leaves. The page runs under a strict content security policy, the fonts and the ONNX runtime ship with the site, and a PerformanceObserver in the page and another in the model's worker count every request to another site. Open a statement, export it, and the count is still 0.

Testing. A generator writes statements for seven invented banks in seven layouts, each with an answer key. The test suite parses all seven and compares every date, amount, balance and description with the key (386 rows; all match and all tie), checks that a misread amount is caught on its own row and nowhere else, that signs come from the balance when the statement prints none, and opens every export with an independent reader.

Why Does Open Innovation Matter?

A bank statement is about as private as a document gets. The only way to put a capable reader next to it without sending it somewhere is to run the model where the document already is, and that only works with open weights. Gemma 4 is Apache 2.0, so a static web page can hand it to anyone's browser: no API key, no account, no bill that grows with every page read. A closed API would have made the whole idea impossible, because the first step would be uploading the statement.

Open code matters for the same reason. "We never see your data" is a sentence every converter puts on its homepage. Here you can read the code and watch the request counter while you work.

And it keeps the business honest. With no inference bill, the core can stay free for good. What people would pay for is time saved, not server time.

Prize Categories

Best Use of Gemma: Gemma 4 E2B reads scanned statements and suggests categories, entirely in the browser on WebGPU, with every figure it reads checked by the statement's own arithmetic.

Would today's AI have caught Ariane 5? I rebuilt 26 of history's costliest bugs to find out

Kai Chen — Fri, 09 Oct 2026 09:35:12 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

Hindsight asks a simple question: if you had shown today's models the code behind history's most expensive software failures, the day before it shipped, would they have caught the bug?

I spent the last few weeks reading accident reports for a series of short animated explainers about famous failures: the Ariane 5 inertial reference software, the Patriot battery at Dhahran, Knight Capital's 45 minutes, the CrowdStrike channel file, the spreadsheet behind Reinhart and Rogoff. Every report ends the same way: the defect was small, local and visible in a few lines, and nobody saw it in time. That made me curious whether a code review by a model would have changed anything.

The benchmark has 26 cases. Each is a documented failure with a published root cause (an inquiry board, a GAO or SEC report, a vendor post-mortem or a peer-reviewed paper), rebuilt as a minimal artifact: a function, a shell script, an Excel formula, an RTOS configuration. No names, no dates. The model gets the artifact and one line about the system, and answers in a fixed format: VERDICT: DEFECT / NO_DEFECT, what fails, the trigger, the consequence. A judge model compares the answer with the published root cause.

Every case comes in four forms, which are the four tasks of the benchmark:

Task	What the model sees	What it measures
Original	the reconstruction in its original setting (Ada on a launcher, Pascal on a radiotherapy machine)	can it catch the failure?
Disguised	the same mechanism moved to another domain, language and naming (the Ariane conversion becomes a Python drone telemetry packer)	is it reasoning, or recognising a famous story?
Disguised + context	the disguised artifact plus the one operating fact that triggered the real failure ("the new drone flies five times faster")	how much does knowing the environment help?
Fixed	the original artifact after the fix that was applied	does it raise false alarms on code that is now correct?

The gap between Original and Disguised is a memorisation signal. The gap between Disguised and Disguised + context measures something the accident reports keep returning to: much of this code was correct for the system it was written for, and failed in a new environment.

One case, four forms

Here is the Ariane 5 case. The original form is close to what flew in 1996: a 64-bit float converted to a 16-bit integer, protected on one variable and not on the other, because analysis of the old rocket's trajectory showed it stayed small.

procedure Update_Alignment is
   HB_Bus : Integer_16;   -- horizontal bias, 16-bit field on the data bus
   VB_Bus : Integer_16;   -- vertical bias
begin
   Horizontal_Bias := Compute_Horizontal_Bias (Horizontal_Velocity);  -- Long_Float
   Vertical_Bias   := Compute_Vertical_Bias (Vertical_Velocity);      -- Long_Float

   VB_Bus := Integer_16 (Saturate (Vertical_Bias, -32_768.0, 32_767.0));
   HB_Bus := Integer_16 (Horizontal_Bias);
   --  no saturation on HB: analysis of the trajectory showed it stays small

   Bus.Put (Field => VB, Value => VB_Bus);
   Bus.Put (Field => HB, Value => HB_Bus);
end Update_Alignment;   -- no exception handler in this procedure

The disguised form keeps the mechanism and changes everything a model could pattern-match on: a cargo drone, Python, a radio packet, millimetres instead of velocities.

def pack_drift(nav):
    # radio link fields are signed 16-bit integers (struct code 'h')
    alt_drift = clamp(int(nav.vertical_drift_mm), -32768, 32767)
    lat_drift = int(nav.lateral_drift_mm)   # analysis showed lateral drift stays small
    return struct.pack('<hh', alt_drift, lat_drift)

def flight_loop(nav, radio):
    while True:
        nav.update()
        radio.send(pack_drift(nav))

The context form adds the one sentence the Ariane 5 team did not connect to this code: "the new drone cruises about five times faster than the quadcopter, so its lateral drift estimate during the climb reaches 40,000 to 60,000 mm." And the fixed form is the Ada original with Horizontal_Bias saturated as well, so a model that still says "this will overflow" is now wrong.

How an answer is scored

A case counts as caught only if the model says DEFECT and the judge confirms that the answer names the same mechanism and trigger as the published root cause. "This conversion could overflow" is not enough for Ariane 5; the answer has to say that the horizontal bias goes past 32,767 and that the unhandled exception stops the unit. A correct verdict with the wrong reason is a miss.

On the fixed versions the rule flips: an answer passes unless it claims that the historical failure will still happen. Raising some other concern is allowed, and I report separately how often models flag something on correct code.

The judge is Kaggle's default evaluation model. To check it, I re-read 64 answers by hand: the 44 disguised answers from the first nine models that said DEFECT but were marked as misses, plus 20 random passes. I agreed with the judge on 61. All three disagreements were the judge being too strict, so if anything the scores below are slightly low.

Models Tested

Thirteen models from six vendors, large and small. Ranking them was half the point; the other half was seeing whether size and vendor change what kind of bug gets missed:

Anthropic: Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 4.5
OpenAI: GPT-6.1 Sol, GPT-5.4 mini, gpt-oss-120b (open weights)
Google: Gemini 3.1 Pro (preview), Gemini 3.8 Flash, Gemini 3.7 Flash, Gemma 4 31B (open weights)
xAI: Grok 4.20 (reasoning)
DeepSeek-R1 and Qwen3 Coder 480B (open weights)

Every model gets the same prompt with the platform's default sampling settings and a 4,000-token output cap, one case per fresh chat.

Findings

The famous bugs are solved, even in disguise

Eight cases were caught by all thirteen models in their disguised form: Ariane 5, the Patriot clock, Mars Climate Orbiter, the Boeing 787 counter, the Azure leap-day certificate, the AT&T switch crash, CrowdStrike and the Reinhart-Rogoff spreadsheet. Ten more were caught by at least ten of the thirteen. A Python drone packer with an unclamped 16-bit field does not fool anyone any more. These are bugs you can see by reading the code once: a narrowing conversion, a counter that wraps, a quantity in the wrong unit, a range that stops a few rows short.

What they miss only shows up when the code runs

Seven cases were caught by eight of the thirteen models or fewer:

Case	Caught	What you have to imagine
Mars Polar Lander, 1999	46%	a vibration spike during leg deployment, latched as "touchdown" long before the ground
Vancouver Stock Exchange index, 1983	46%	a truncation to three decimals, repeated on every trade for 22 months
Cloudflare WAF, 2019	46%	one regex meeting a long input and backtracking until the CPU is gone
England's COVID test results, 2020	46%	an export format that stops at 65,536 rows
Amazon S3, 2017	54%	a maintenance command removing far more capacity than the operator meant
Schiaparelli, 2016	54%	an inertial unit that saturates for one second while the parachute opens
London Whale, 2012	62%	a volatility that is divided by a sum where it should be an average

None of these is hard to read. Each one needs you to run the code in your head under a condition that is not on the page: a long input, a vibration, the 22nd month, the 65,537th row. That is exactly what the accident reports say the engineers failed to do, and it is the part the models still struggle with.

Right verdict, wrong reason

Across the 338 disguised answers the models said DEFECT 97% of the time. Of the 63 misses, 54 still said DEFECT: they found a problem, just not the one that brought the system down. Some of the wrong reasons are reasonable review comments. Some are invented.

On the Cloudflare regex, Claude Haiku 4.5 warned about SQL injection ("DROP TABLE users;--"). The regex never touches a database; the problem is that it can take exponential time.
On the Vancouver index, Gemini 3.1 Pro blamed floating point: "0.58 * 100 evaluates to 57.99999999999999". True, and a one-cent error, while truncating on every update took half the value of the index.
On England's spreadsheet export, Gemini 3.8 Flash said the export call "accepts only a single argument" and will crash. It takes a file path just fine, and the answer never mentions the row limit of the .xls format.
On Mars Polar Lander, GPT-6.1 Sol found the latch that is never cleared, but blamed a previous trip instead of the vibration on the way down: the right line of code with the wrong story.

This is why a DEFECT verdict on its own means very little here, and why the benchmark scores the mechanism and the trigger, not the verdict.

My reconstructions had bugs of their own

Reading the misses by hand, I found that three of my artifacts contained a second, unintended defect, and some models reported that one instead of the historical one: the Cloudflare regex also rejects a valid single comparison, the London Whale sheet divides by zero when both weeks are empty, and the S3 drain script deletes nodes without waiting for the evictions to finish. Those answers are correct review comments and still count as misses, because they are not what happened. When you rebuild a failure, you rebuild more than one failure. I have left the cases as they were run and listed the extra defects in the benchmark notes.

Memory, context and false alarms

Averaged over all thirteen models, the four tasks score 89% (original), 81% (disguised), 89% (disguised with context) and 96% (fixed version, no false alarm).

Memory helps less than I expected, and not where I expected. The memorisation gap, original minus disguised, is eight points on average. It is largest for DeepSeek-R1 (19 points), gpt-oss-120b and Gemma 4 31B (15 each), and zero or below for Claude Sonnet 5.5, Claude Opus 5.5 and Qwen3 Coder. I also counted how often an answer to the original form names the real incident ("this is the Ariane 5 failure"). Opus did that on 81% of cases, Gemini 3.1 Pro on 58%, Sonnet on 46%, and nearly every other model on 8% or less. The models that recognise the stories most often are the ones that lose nothing when the story is taken away, so recognising a case and depending on that recognition are two different things. The gap belongs to the mid-sized and open models.

One sentence of context is worth as much as the original setting. Adding the operating fact to the disguised artifact brings the average from 81% back to 89%, level with the original. The cases it rescues are exactly the hard ones from the table above: the spreadsheet export (7 models went from miss to catch), the regex, the lander and the parachute IMU (6 each). DeepSeek-R1 gains 23 points, GPT-5.4 mini and Gemma 15. Grok 4.20 is the exception: it lost three cases when given the context (Zune, S3 and the London Whale sheet), and the Zune case lost five models in total, because the hint about the last day of a leap year pulled answers toward the date arithmetic and away from the loop that never ends.

Fixed code is rarely condemned, but often criticised. Only 15 of 338 answers claimed the historical failure would still happen on the fixed version, and six of those were the same case: Schiaparelli, where the fix drops the one-second hold on the saturated rate and adds a plausibility check, and six models still argued that the altitude could go negative. But models flagged some defect on 44% of the fixed artifacts. Claude Haiku 4.5 did so on 96% of them, GPT-5.4 mini on 88%, gpt-oss-120b on 81%; Sonnet 5.5 never did and Opus 5.5 once. For a code reviewer that difference matters as much as the catch rate: a tool that objects to everything teaches people to stop reading its comments.

My Benchmark

The benchmark on Kaggle: Hindsight: Catching History's Costliest Bugs, with the leaderboard for all thirteen models.

The four tasks, each with the reconstructions, the sources for every case and the per-model runs:

Aurora Odds: should you go outside and look up right now?

Kai Chen — Wed, 07 Oct 2026 01:36:32 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

What I Built

On 10 May 2024 the northern lights reached Florida, northern India and the Canary Islands. Millions of people saw them. Many more only read about it the next morning, because the alerts on their phones said "G5" and "Kp 9" and nothing about their own sky.

That is the gap I wanted to close. Every aurora app I tried answers a forecaster's question (how disturbed is the magnetic field?) when the question people actually ask is a practical one: is it worth putting my shoes on and going outside, here, right now?

Aurora Odds answers exactly that. Pick your place (or let the browser find it) and you get one verdict for your sky, Go outside now, Bring your phone, Not yet: it's still light out, or Stay in, the sky is quiet, backed by four numbers:

the chance the aurora is bright enough for your eyes in the next hour, and for a phone camera, which sees it from much further away;
whether it is dark enough where you are, and when it will be;
how much of your sky is cloud;
which way to face, and how high to look: "lower edge about 8° above the horizon, top near 25°, three fists at arm's length".

There is also a red night-vision mode, because the whole point is to use it outside in the dark without wrecking your eyes' adaptation, and a "tell me when to go out" switch: leave the tab open and it chimes once it is dark, mostly clear and the odds for your eyes pass 40%. The screen can stay in your pocket until then.

The forecast comes from TabPFN, an open-weights tabular foundation model, reading the solar wind that is on its way to Earth.

Demo

Live: https://luoy16002-svg.github.io/aurora-odds/

The page is most fun during a storm, and storms are rare, so it has a replay mode that runs the same model on the solar wind of a real night: London, 10 May 2024 and Chicago, 10-11 October 2024. Clouds in a replay come from Open-Meteo's historical archive for that hour.

Code

luoy16002-svg / aurora-odds

Should you go outside and look up right now? A live aurora nowcast for your exact sky, made by TabPFN reading the solar wind.

Aurora Odds

Should you go outside and look up right now? Aurora Odds answers that for your exact sky: the chance the aurora is bright enough for your eyes or your phone camera in the next hour, whether it is dark and clear enough and which way to face and how high to look.

Live page: https://luoy16002-svg.github.io/aurora-odds/ · storm replays: London, 10 May 2024, Chicago, 10-11 October 2024

The forecast is made by TabPFN, an open-weights tabular foundation model reading the solar wind measured at the L1 point 1.5 million km upstream. It is not trained on this problem: it reads 2,000 labelled half-hours from 1998 to 2019 in context (8,000 in the evaluation below) and returns a full probability distribution for the coming hour's geomagnetic activity. One distribution answers every latitude.

How it works

Upstream measurements. NOAA's real-time solar wind feed (magnetic field and plasma from the…

View on GitHub

How I Built It

The solar wind gives you a head start

Spacecraft parked at the L1 point, 1.5 million kilometres sunward, measure the solar wind every minute. That wind takes 20 to 80 minutes to reach Earth, so part of the next hour is already measured before it arrives. The most important number is Bz, the north-south direction of the wind's magnetic field. When it points south it connects to Earth's field and pours energy in; when it points north, very little gets through, however fast the wind blows.

I wanted the model to see the world exactly as it can be seen at one moment, so every input is built on an arrival-time axis: each sample is shifted by its own travel time, and at time t the model is only allowed to use samples that had already been measured at L1 by t. That gives four windows: what is measured but still on its way, the last 30 minutes, 30 to 90 minutes ago, and 90 to 180 minutes ago. In each I summarise Bz, the Newell coupling function (a standard estimate of how much energy the wind is delivering), speed, density and pressure, plus the season and time of day. The same code builds 28 years of history from NASA's OMNI data and the live features from NOAA's real-time feed, so there is no training/serving skew to debug at 2 a.m. during a storm.

The target is the highest Hp30 in the coming hour. Hp30 is GFZ Potsdam's half-hourly version of the Kp index: same scale, finer timing, and open-ended, so a superstorm is not squashed at 9.

TabPFN reads history instead of training on it

TabPFN is a transformer pretrained on millions of synthetic tabular problems. You don't train it on your data: you hand it a table of labelled examples (the context) and it predicts new rows in a single forward pass, like in-context learning in a language model.

I gave it 8,000 half-hours from 1998 to 2019 as context (the live page uses 2,000; more on that below). Storms are rare (the next hour reaches Hp30 6 only about 1.2% of the time), so a uniform sample shows the model very few of the moments that matter. I tried a stratified context that over-represents storms and then corrects the predicted distribution back to the real storm frequency. On the 2020-2022 validation years the stratified context won (mean Brier skill 0.46 against 0.44 for Hp30 ≥ 5 to 7), so that's what runs.

The part I like most: TabPFN's regressor returns a full probability distribution over the next hour's Hp30, not a single number. So one forward pass answers every latitude at once. Tromsø needs almost nothing, Edinburgh needs about Kp 5 to see it with the naked eye, London about 7.5. Each of those is just P(Hp30 ≥ k) read from the same distribution.

Did it work?

I tested on 2023 to 2026, years the model never saw, which include the solar maximum, the May 2024 superstorm and the October 2024 storm. I compared it with:

LightGBM trained on all 370,000 half-hours from 1998 to 2019 (a classifier per threshold, the usual way to do this);
persistence: "the next hour will look like the last half-hour", using the real index with zero delay, which no live app actually has.

Brier skill score (AUC)	Hp30 ≥ 4	Hp30 ≥ 5	Hp30 ≥ 6	Hp30 ≥ 7	Hp30 ≥ 8
TabPFN, 8,000-row context	0.56 (0.951)	0.54 (0.975)	0.52 (0.987)	0.58 (0.993)	0.53 (0.990)
TabPFN, 2,000-row context (the live page)	0.54 (0.948)	0.52 (0.973)	0.51 (0.986)	0.56 (0.992)	0.50 (0.992)
LightGBM, trained on 370,431 rows	0.56 (0.951)	0.54 (0.974)	0.51 (0.987)	0.46 (0.989)	0.34 (0.982)
Last half-hour's index (oracle)	0.34 (0.790)	0.33 (0.777)	0.34 (0.776)	0.45 (0.811)	0.44 (0.804)

61,420 half-hours from January 2023 to September 2026. Storm half-hours at each level: 9,345, 3,458, 1,201, 485, 166. Brier skill is relative to always forecasting the long-run frequency; 0 means no skill.

For moderate activity, TabPFN and LightGBM are level. For the rare, strong storms, the ones that bring the aurora to Edinburgh, Chicago or London, TabPFN is clearly better: a Brier skill of 0.58 against 0.46 at Hp30 ≥ 7 and 0.53 against 0.34 at Hp30 ≥ 8. LightGBM saw 46 times more rows, but strong storms are a sliver of them and a tree has very little to split on out there. My guess is that TabPFN's prior, learned from millions of synthetic problems, degrades more gracefully in the tail; either way, it's the tail that decides whether someone in London goes outside.

The live page runs the 2,000-row context, so a free 4-core GitHub runner finishes in about a minute. That costs about 0.02 of skill.

It is not magic. On 10 May 2024 the index climbed to about 6 in the early afternoon while the wind measured upstream still looked calm (a weak field of about 3 nT, barely southward). A model that only sees the wind had nothing to react to and sat at Kp 2 to 3 until the storm's shock arrived at 17:10 UTC; after that it was in the right place within half an hour. It also over-forecasts a little at moderate levels: in the reliability chart the curves for Hp30 ≥ 5 and 6 sit slightly below the diagonal. For a tool whose only job is to tell you when to put your shoes on, I'm fine with erring that way.

From a number to a sky

The rest is geometry, and it's where most aurora apps stop short.

Magnetic latitude. The aurora follows Earth's magnetic field, not lines of latitude, and a simple tilted-dipole model puts Europe several degrees too far north. I precomputed corrected (AACGM-v2) magnetic latitude on a 2° grid and interpolate it in the browser. Edinburgh comes out at 53°, Minneapolis at 54°: that is why Minnesota sees the aurora more often than Scotland despite being 11° further south.
What you need. The auroral oval's edge sits near 66.5° magnetic latitude when the field is quiet and moves about 2° equatorward per Kp step. Low on the poleward horizon it is visible from a few degrees further away, and a phone's night mode picks it up further still. I checked the margins against what observers report for Tromsø, Edinburgh, Minneapolis, London and the May 2024 storm.
How high to look. Light is emitted 110 to 300 km up. From the distance between you and the oval, a few lines of spherical geometry give the elevation of its lower edge and top, and because the oval is a band at fixed magnetic latitude, it shows up as an arch, highest toward the magnetic pole.
Dark and clear. The sun's altitude comes from a standard solar position formula (dark enough below −12°), and cloud cover from Open-Meteo.

A GitHub Actions job reruns the model every ten minutes on a CPU runner and publishes the result with the static page.

Why Does Open Innovation Matter?

Three things in this project only work because the pieces are open.

The model can be hammered for free, at exactly the wrong moment. Aurora traffic is the spikiest traffic there is: nobody looks for months, then a storm hits and everyone looks within the same ten minutes. Because TabPFN's weights are open, the model runs once every ten minutes in a free CI job and writes a small JSON file that a CDN serves to everyone. A million visitors cost the same as one. There is no API key in a secrets store, no per-call bill and no rate limit that kicks in during the one night that matters.

I could change what the model returns, not just what I send it. The label-shift correction works on TabPFN's full predictive histogram: I re-weight its bars by how much more often storms appear in the context than in reality. That needs the raw distribution. An endpoint that returns one number would have made the stratified context unusable, and the "one distribution, every latitude" trick impossible.

Every claim in this post can be checked. The solar wind history is NASA's OMNI data, the index is GFZ's Hp30 (CC BY 4.0), the live feed is NOAA's and the clouds are Open-Meteo's. With open weights on top, python pipeline/evaluate.py reproduces the numbers above on a laptop or a single consumer GPU. If you think my context is badly chosen, or that a different set of features would do better, you can try it in an afternoon and show me.

Prize Categories

Best Use of TabPFN: TabPFN is the forecaster. It reads the labelled history in context and its full predictive distribution turns into the chance of seeing the aurora at every latitude.