DEV Community: Bowen Rui

chaptermill: a podcast's back catalog to show notes for 3¢ an audio hour, on an old PC I kept crashing on purpose

Bowen Rui — Thu, 15 Oct 2026 20:19:46 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

Most podcasts that have been running for a few years have the same problem: hundreds of old episodes with no transcript, no chapters and no show notes worth reading. That archive is what new listeners stumble on through search, and it is the part nobody wants to pay per-hour SaaS prices to fix.

chaptermill takes a podcast's RSS feed and, on a machine you already own, turns the back catalog into the files a producer actually delivers:

a timestamped transcript (SRT, VTT, JSON),
chapters as a Podcasting 2.0 chapters.json and as YouTube chapter lines,
2–4 pull quotes, each checked word for word against the transcript before it is accepted,
Markdown show notes with a summary, topics and a credit line.

Who pays for it: freelance podcast producers (show notes are a standard deliverable) and independent shows with a big archive. The hosted tools price by the audio hour. On their pricing pages today, Castmagic's Hobby plan is $19/month billed annually for 30 transcribed hours in total, and its Team plan, 400 hours, is $139/month billed annually; Podsqueeze's Agency Lite plan is $89/month for 600 minutes per month. A 300-episode archive of half-hour shows is 150 hours. With chaptermill, at the rate of my demo run, the same archive costs about $4.40 in open-weight model calls and a day of runtime on a 2015-era desktop. The side hustle writes itself: a flat-fee "archive pass" for small shows, where the marginal cost is cents. For scale: Hacker Public Radio, whose episodes I used for the demo, numbers its episodes past 4,700. At the average length of the nine I processed (28.5 minutes) that is roughly 2,250 hours, or about $65 of model calls at my demo run's rate. That is an estimate from a sample of nine, not a quote.

To be fair to the incumbents: Apple Podcasts and Spotify now generate chapters for some shows inside their own apps. That helps listeners on those apps. It does not give the producer a file to put in the feed, on YouTube or on the show's website, and it does not touch the archive in one go.

Demo

Live: https://bowen1314.github.io/chaptermill/

Nine recent episodes of Hacker Public Radio (a community podcast released under CC BY-SA 4.0), processed on my home server (one run, plus a re-run of two episodes that I explain below), with nothing edited by hand. Pick an episode; press a chapter or a quote and the player jumps there; open the full transcript and the current line follows the audio. Every file is downloadable.

Further down the page is the part I am proudest of: the report from a second run during which a script SIGKILLed the worker 9 times and the Temporal server once, while the worker injected model-API 503s, cut-off downloads and failing tool calls on purpose.

Code

Bowen1314 / chaptermill

Turn a podcast back catalog into transcripts, chapters, verified pull quotes and show notes with open-weight models; Temporal makes it survive crashes.

chaptermill

Point it at a podcast RSS feed and leave it running on an ordinary PC. It turns the back catalog into transcripts, chapters, pull quotes that are checked word for word against the transcript, and show notes ready to paste (the demo run did 4.3 hours of audio in 44 minutes on a 2015-era desktop). Open-weight models do the work. Temporal makes it durable: a crash, a dropped connection or a rate-limited model API means it carries on from where it stopped, without redoing finished work or re-buying model replies it already has.

Live demo: https://bowen1314.github.io/chaptermill/ (nine Hacker Public Radio episodes, plus the report from a run where I kept killing the worker and the Temporal server)

Built for the DEV Hacktoberfest Open-Source AI Challenge, week 2 (Side Hustle), Temporal category.

Who it is for

Freelance podcast producers who write show notes for clients, and independent shows sitting on…

View on GitHub

MIT licensed. About 1,900 lines of Python plus 560 lines of tests, including workflow tests on Temporal's time-skipping test server.

How I Built It

Three open pieces do all the work:

Piece	What it does	License
faster-whisper `base.en`	transcription, locally, on 2 CPU threads	MIT
Qwen3-235B-A22B-Instruct-2507	the show-notes agent, through Nebius Token Factory's OpenAI-compatible API	Apache-2.0
Temporal (dev server + Python SDK)	durable orchestration, retries, signals, queries	MIT

The machine is a shared home server: an Intel i3-6100 (2 cores), 3.8 GB of RAM and one spinning disk, also running a dozen other services.

The shape of it

A CatalogWorkflow reads the feed and starts one child EpisodeWorkflow per episode, two at a time, inside a dollar budget. Each episode workflow downloads the audio, cuts it into 5-minute chunks (one activity per chunk, on its own task queue so Whisper never starves the rest), merges the transcript, then runs the agent loop and writes the files.

The agent gets a one-line-per-minute outline and three tools: read_transcript(start, end), search_transcript(query) and submit_notes(...). Every model call and every tool call is a separate Temporal activity, so each one has its own timeout and retry policy, and a crash in turn 6 resumes at turn 6:

for _turn in range(agent.MAX_TURNS):
    await self._wait_for_cap()            # durable wait for a raise_cap signal if over budget
    resp = await workflow.execute_activity_method(
        Activities.llm_turn,
        LlmRequest(model=inp.model, messages=messages, tools=agent.TOOLS, ...),
        start_to_close_timeout=timedelta(minutes=3),
        heartbeat_timeout=timedelta(seconds=30),   # a dead worker is noticed in 30 s
        retry_policy=LLM_RETRY,                    # 429/5xx retried, other 4xx fail fast
    )
    messages.append(resp.message)
    for call in resp.message.get("tool_calls") or []:
        if call["function"]["name"] == "submit_notes":
            v = await workflow.execute_activity_method(Activities.validate_notes, ...)
            out = "Accepted." if v.ok else "Rejected. Fix these and call submit_notes again:\n- " + "\n- ".join(v.errors)
        ...  # read_transcript / search_transcript: one activity each
        messages.append({"role": "tool", "tool_call_id": call["id"], "content": out})

Audio, transcripts and chunk results stay on disk; workflow history carries file paths and the agent's conversation, nothing bigger.

The model is checked, not trusted

submit_notes does not end the loop. A validator compares the submission with the transcript, and if anything is off, the error goes back to the model as the tool result. In the demo run it rejected 4 submissions; the model's next submission fixed each of them. Two of them are worth showing, because they are opposite failures:

HPR4749: the model's first quote silently cut a sentence out of the middle ("And from the website, I'm gonna quote it here.") and stitched the rest together. Rejected; the resubmission quoted the host exactly.
HPR4747: the model wrote "in place of cat", which is what the host said (the Unix command). Whisper had transcribed it as "CAD". The validator rejected the correct word, and the final quote says "CAD".

That second one is the honest limit of this design: quotes are guaranteed to match the transcript, not the audio. A custom vocabulary for Whisper is the obvious next step.

Making it durable, for real

Temporal's AI cookbook patterns map almost one to one onto this pipeline:

Retries in one place. The HTTP client's own retries are off. Errors are classified: 408/409/429/5xx are retryable, other 4xx fail fast, and a Retry-After header becomes the activity's next_retry_delay.
Idempotent model calls. Each reply is written to a ledger and to a cache keyed by a hash of the request before the activity returns, so if the worker dies a millisecond later, the retry reads the cache instead of buying the reply again.
Chunks as checkpoints. A finished chunk is a file. Whisper runs at temperature 0, so a re-run chunk produces the same text.
Heartbeats on every long activity, including a side thread that heartbeats while the CPU activity is stuck loading the model.
Budget as a durable wait. Episodes that would go over the run's budget wait in workflow.wait_condition until a raise_budget signal arrives, for as long as it takes; pause/resume signals and a progress query come for free.

Then I tried to break it. scripts/chaos.py started a three-episode catalog and, every 45 to 110 seconds, SIGKILLed the worker, kept it down for 5 to 20 seconds and started it again; on the third kill it took the Temporal server down too. On top of that, the worker failed 30% of model calls with a fake 503, cut off 50% of audio downloads and broke 15% of transcript tool calls, each only on an activity's first two attempts. The result:


Kills	9 SIGKILLs of the worker at random moments, one of them together with the Temporal server
Episodes finished	3 of 3, in 12.5 minutes, with no manual intervention
Transcripts vs. the undisturbed run	identical, segment for segment (584 segments)
Activity attempts	88 for 67 activities: 8 retried transcription chunks, 8 retried model calls, 3 retried downloads, 2 retried tool calls
Model replies received	18, none bought twice
Requests in flight at a kill	1: its reply died with the process and the retry asked again (the provider may have billed both)
Fake 503s absorbed	6
Model cost	$0.021

Each kill cost at most the one chunk or model call in progress. The unluckiest chunk, in HPR4740, was killed three times and finished on its fourth attempt; nothing before it was redone.

The box itself added chaos I had not planned. During the clean run, another service on the same disk was writing about 16 MB/s, and Linux reported I/O pressure between 40% and 87%. The Temporal dev server keeps everything in SQLite, so every history write queued behind that disk, and a few tasks were handed to my worker after the worker had stopped waiting for them. Temporal's answer was a timeout and a retry; the run finished anyway, with 20 workflow-task timeouts in its history. Halfway through I restarted the server with SQLite in WAL mode and synchronous=normal (still safe when the process is killed, which is the failure I care about here): before the switch 19 of 78 workflow tasks timed out, after it 1 of 222. Not a controlled experiment, but a big enough difference that the chaos script now starts the server that way. I also learned not to fsync inside an async activity: on that disk a single fsync blocked the event loop the workflows share for seconds. Disk-heavy activities now run as sync activities on a thread pool.

Numbers from the demo run


Episodes	9 Hacker Public Radio episodes, 4.3 h of audio, 38,318 words
Wall clock	44 minutes, two episodes at a time
Transcription	26 minutes for 4.3 h of audio (9.8× real time), Whisper `base.en` on 2 CPU threads
Agent	72 turns, 91 tool calls, 4 rejected submissions, all fixed
Model replies paid for	74, none bought twice: $0.125 in total, $0.029 per audio hour
Difference from HPR's own transcripts	12.6% of words (HPR's are also made with Whisper and not proofread, so this is two machine transcripts disagreeing, not an error rate)

One honest miss: the first run finished 7 of 9. On the two longest interviews the agent spent all ten of its turns reading the transcript window by window and never called submit_notes. I added a "three turns left, submit now" message and re-ran just those two. Every transcript chunk and the first nine model replies of each came straight from cache; only the last turn was new, at about a third of a cent each. The $0.125 above includes the wasted turns.

Why Does Open Innovation Matter?

Cost. The expensive half of this job, turning audio into text, costs nothing per minute when the model runs on your own CPU. OpenAI lists its Whisper API at an estimated $0.006 per minute, $0.36 per audio hour; my entire language-model bill was $0.029 per audio hour. Open weights move the dominant cost from "per minute, forever" to an old PC's spare time.

Privacy. The audio never leaves the machine. Only transcript text goes to the model endpoint, and because the model is open-weight, a producer whose clients will not allow even that can point chaptermill at their own server. (I want to be precise: for the demo I used a hosted endpoint; I did not run the 235B model at home.)

Reproducibility and no lock-in. The Whisper weights are a file on my disk that nobody can update or retire under me, so running an episode again gives the same transcript; the chaos run shows it, with transcripts identical segment for segment to the undisturbed run. The agent talks to any OpenAI-compatible server, so switching model or provider is a config change, not a rewrite. I did try Gemma 3 27B first and it ignored the tool definitions on that endpoint, so "any" means any model whose tool calling works.

Fine-tuning for a niche. The "cat"/"CAD" miss above is exactly the kind of error open weights let you fix yourself. A producer whose shows are full of jargon (Unix commands, drug names, player names) can fine-tune base.en on a few hours of their own corrected transcripts, or at least hand Whisper a vocabulary prompt, and run the result on the same CPU. A closed transcription API may let you pass a prompt, but you cannot retrain it on your show. I have not done this yet; it is next on the list.

The orchestration is open too. Temporal's dev server ran on the same old box, with a SQLite file, no account and no cloud.

Prize Categories

Temporal. The category asks for an agent that "survives failures, retries flaky tool calls, and picks up where it left off". chaptermill's agent is a Temporal workflow where every model call and tool call is an activity, and the demo page shows the evidence from a run where I killed the worker 9 times and the server once.

Which trees near me are turning this weekend? Forecasting fall color from 6,650 iNaturalist observations with TabPFN

Bowen Rui — Tue, 06 Oct 2026 19:55:14 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

Built with PriorLabs-TabPFN.

What I Built

Fall color doesn't show up on a fixed date. It comes when the nights turn cold, and every fall turns cold on its own schedule. Foliage maps can tell you that a region is "near peak". They can't tell you whether the red maples on your usual walk will be red on Saturday, or whether you should wait another week.

peakweek answers that question for the trees around you. You type a town in the US Northeast or Mid-Atlantic, wait about a minute, and get one card for the coming weekend:

for each of 8 common fall-color trees (red maple, sugar maple, sweetgum, northern red oak, American beech, black gum, sassafras, Norway maple), the chance that a tree of that species you come across will be green, in color, or bare;
a verdict per species (go, starting, wait, late) and the day in the next two weeks with the best chance of color;
one line on how to recognize each tree, so you know which one you're looking at once you're out there;
one plain sentence at the top that tells you what to do.

Here is the headline for New Brunswick, New Jersey, from a forecast made on Tuesday, October 6, for the weekend of October 10–11:

No tree here is in full color yet this weekend, but sassafras, black gums and sugar maples are starting to turn; come back around Mon, Oct 19 for northern red oaks.

And here it is for Burlington, Vermont, from the same day:

This weekend, go see red maples, sugar maples and black gums; come back around Tue, Oct 20 for northern red oaks.

Then the screen gets out of the way. Print card gives you a one-page, black-and-white card to take with you. Done, go outside shrinks the page to that one sentence. The footer asks for one thing after your walk: add what you saw to iNaturalist with the "Leaves" annotation. Those annotations are exactly what the model learns from, so every walk makes next fall's forecast a little better.

It's for anyone who likes to go look at leaves and would rather spend that time outside than refreshing a foliage map.

Demo

This is a real run, searching Burlington, Vermont: the server runs TabPFN on a 4-core CPU with no GPU, and the 100 seconds it spent on this forecast (the first one after the server started) are sped up 10x in the GIF. Full video.

On a phone, Use my location near New Brunswick gives the "not yet" version of the card:

And this is what Print card produces, one black-and-white page:

Try it: bowen1314.github.io/peakweek. A forecast takes about a minute of CPU, so instead of a server there's a GitHub Actions job: every morning it fetches the weather and runs TabPFN once for each 1-degree grid cell of the region that has training data (103 cells), and GitHub Pages serves the cards. The page shows the card for the cell around the place you search, made at that cell's weather point, so it's a 1° approximation (a cell is about 110 × 80 km), and every card says which cell it's for and the day it was made. Before anything is published, the build re-runs a few cells through the ordinary single-point forecast and stops unless the numbers match. For your exact spot, run it locally. The repo also includes saved real forecasts for New Brunswick, Burlington and Pittsburgh, so you can open the full UI without installing the model:

python3 server.py --fixture examples/forecast_new_brunswick.json   # http://127.0.0.1:8770/
python3 -m peakweek --fixture examples/forecast_burlington.json      # the same card, in the terminal

Code

Bowen1314 / peakweek

Which trees near me are turning this weekend? Fall color forecasts from iNaturalist + ERA5 weather with TabPFN

peakweek

Which trees near you are turning color this weekend, and which day should you go out?

Built with PriorLabs-TabPFN. peakweek forecasts fall color for 8 common trees in the US Northeast and Mid-Atlantic from 6,650 iNaturalist community-science observations and this season's weather, using the open TabPFN v2 weights on a plain CPU. The answer is one card you can print, and then you go outside.

The waiting part (about a minute of TabPFN on a 4-core CPU) is sped up in the GIF. Full video: docs/demo.mp4.

Live site

https://bowen1314.github.io/peakweek/: search a town or press Use my location; nothing to install.

Updated daily. Every morning (around 10:20 UTC, about 6:20 in New York) a GitHub Actions job (.github/workflows/pages.yml, scripts/build_site.py) fetches this season's weather from Open-Meteo, runs TabPFN once on the runner's CPU for every grid cell below, and publishes the cards to GitHub Pages…

View on GitHub

The app itself (server, page, card, verdict logic) uses only the Python standard library. The model side needs tabpfn and a CPU build of PyTorch. MIT licensed. The TabPFN v2 weights are licensed separately by Prior Labs.

How I Built It

The data: people photographing trees

On iNaturalist, people can tag a tree photo with what its leaves are doing: Green Leaves, Colored Leaves, or No Live Leaves. That makes it a labeled phenology dataset that nobody had to organize. I pulled every annotated September–November observation of the 8 species in a box covering the US Northeast and Mid-Atlantic (latitude 38.5–47.5, longitude −80.5 to −66.9), from 2018 to 2025.

The API returned 10,877 observations, and most of the work was deciding what to throw away:

dropped	observations
planted or cultivated trees	879
"casual" quality grade	52
obscured coordinates	244
location accurate to worse than 1 km	455
conflicting annotations (every one of them "green" and "colored" on the same tree)	856
no usable leaf annotation	1
no open license (I don't redistribute these)	1,740
kept	6,650

That leaves 3,457 green, 2,846 colored and 347 bare observations, all in data/observations.csv with their iNaturalist ids and license codes.

The features: this year's weather, not just the calendar

The calendar and latitude explain a lot of fall color. The question I cared about is whether this fall's weather adds anything on top of that. So for every observation, the same code that the app uses (peakweek/features.py) computes, from Open-Meteo's ERA5 archive:

chilling degree days since September 1 (the sum of how far each day's mean temperature fell below 20 °C);
frost nights (≤ 0 °C) and cold nights (≤ 5 °C) since September 1;
last week's mean minimum temperature and the last two weeks' mean temperature;
rain over the last 30 days;
how much warmer or colder this fall has been than the same place's other falls;

plus species, latitude, longitude, elevation, day of year and day length. Weather comes per 1-degree grid cell. A finer grid would have needed about 15,000 weighted calls to Open-Meteo, more than the free tier allows in a day. The whole project used about 7,000.

One detail I didn't expect: Open-Meteo's forecast API can also return recent past days, which looked like an easy way to get "this season so far". But for September 2026 it returned only about 50 of the 92 past days I asked for, and its rainfall ran 2.3 mm/day below the archive. A model trained on archive weather would have been reading a different instrument. So the app takes September 1 to yesterday from the same ERA5 archive as the training data, and uses the forecast API only for today and the next 14 days.

The model: TabPFN v2, on a CPU

TabPFN is a tabular foundation model from Prior Labs. You don't train it on your data. At prediction time you show it a set of labeled rows (the "context") along with the rows you want predicted, and it returns class probabilities with no training step at all. That suits this problem well: a few thousand rows, a mix of categorical and numeric features, and an answer where calibrated probabilities matter more than the top class.

I used the TabPFN v2 open weights (Prior-Labs/TabPFN-v2-clf on Hugging Face, 29 MB, Apache 2.0 plus an attribution requirement). They download without an account and run on a CPU. On a 4-core machine with no GPU, under a hard 800 MB memory limit, one forecast (8 species × 15 days = 120 rows) takes 51–58 seconds once the server is warm (the first forecast after it starts took 99 seconds in the demo recording), using about 670 MB of memory.

The interesting choice is which rows go into the context, since on a CPU, inside 800 MB, about 1,000 rows is the practical limit. I tried four strategies on a validation year:

One context per species was the obvious idea, and it broke in a TabPFN-specific way. Some species have almost no "bare" rows (sweetgum has 1 in 180), and when a class is missing from the context, TabPFN gives it near-zero probability. A single bare tree of such a species then costs a fortune in log loss: 0.577 overall, against 0.436 for one shared context. Adding a little mass to the missing classes brought it down to 0.461, still worse.
Nearest neighbours in place and week (a separate context for each place and week) was 24 times slower and not measurably better (−0.002 log loss, 95% CI −0.028 to +0.022).
One shared context of 1,000 rows, sampled evenly across species × leaf state, did best.
Averaging three such contexts (different random samples) beat a single one in 97% of paired bootstrap resamples. That's the final setup: 3 contexts × 4 estimators.

Is it any good? One shot at a held-out year

I split by year, so the model never sees the fall it's being tested on. I chose every setting above on 2024 (training on 2018–2023), wrote the configuration to eval/locked.json, and then scored 2025 once (training on 2018–2024; 1,725 observations). Lower is better for log loss and Brier score:

model, 2025 test	log loss	Brier	accuracy
TabPFN v2, all features	0.498	0.285	0.805
gradient-boosted trees (sklearn), all features	0.505	0.291	0.802
TabPFN v2, calendar only (species, place, day of year)	0.516	0.291	0.800
logistic regression, all features	0.519	0.289	0.805
logistic regression, species + day of year + latitude	0.552	0.307	0.795
climatology (how often each species was colored that week in past years)	0.581	0.345	0.752

What I take from it:

TabPFN beat the baseline I committed to in advance. On the 2024 validation year, logistic regression on the same features was the strongest baseline, and it tied TabPFN almost exactly (0.4305 against 0.4307). On 2025, TabPFN was better by 0.021 log loss: 95% paired-bootstrap CI −0.035 to −0.007 when resampling observations, and −0.041 to −0.003 when resampling grid-cell × week clusters.
This fall's weather matters. The same TabPFN given only species, place and date was 0.018 worse (CI −0.026 to −0.011). Knowing how cold the nights have been beats knowing the date.
It's not a clean sweep. Gradient-boosted trees came within 0.007 of TabPFN, which is not a significant difference. Per species, TabPFN was best for red maple, sugar maple and American beech, while simpler models did better on northern red oak, black gum, sweetgum and sassafras (45–164 test observations each). Accuracy is about 0.80 for every decent model, so the gain is in the probabilities, not in the top guess.
Years differ more than models do. Every model scored worse on 2025 than on 2024 (logistic regression went from 0.431 to 0.519).

I also checked the live pipeline on this fall. There were 99 annotated observations from September 26 to October 5, 2026 that passed the same quality filters (77 green, 22 colored), all made after the training data ends. Logistic regression scored 0.436 and TabPFN 0.454. That's too few observations to rank the models (CI on the difference −0.011 to +0.046), but it shows the app's probabilities are sensible on data nobody has seen. Every run, including the ones that lost, is in eval/RESULTS.md.

From probabilities to "go outside"

The card never changes the model's numbers. It only summarizes them, with rules simple enough to print on the card:

verdict	rule (weekend average)
late	chance of bare branches ≥ 35%
go	chance of color ≥ 50%
late	color is already fading: falling since earlier in the window while bare branches rise
starting	chance of color ≥ 25%
wait	otherwise, with the day color is most likely

I also had to settle on wording. "About 4 in 10 sugar maples you find this weekend should show fall color" is honest, because that's what the model predicts: one tree you come across, not the share of the canopy.

The first version of the card had a problem you'd only catch by reading it: it told Burlington, Vermont, to go see sweetgum this weekend. The weather side of the forecast wasn't the issue; the model simply has no idea where trees grow. iNaturalist has no research-grade sweetgum records at all within 50 km of Burlington. So for each place, the app now makes one more keyless call to iNaturalist and counts research-grade observations of each tree within 50 km. A tree with fewer than 5 stays on the card with its numbers, but it's listed last, dimmed, marked "Rarely recorded within 50 km", and left out of the headline. In Burlington that moves sweetgum (0 records) and sassafras (3) to the bottom. In New Brunswick, every one of the eight trees has at least 541 records nearby.

What it can't do yet

It's better at "not yet" than at "too late". People rarely photograph bare trees: only 347 of 6,650 observations are bare. So the chance of bare branches stays small even late in the window, and in early October the best day is often the last day of the forecast. "Late" is rare on the card.
The labels come from what people chose to photograph. A brilliant maple gets photographed more often than a dull one.
The data is recent. 2024 and 2025 make up 65% of the rows, and 2018 has only 53.
Weather is per 1-degree cell, so a cold hollow and a sunny ridge in the same cell get the same weather (elevation is still a feature).

Why Does Open Innovation Matter?

For peakweek, "open" isn't a slogan. Every layer of the project depends on something open:

Open weights mean no gatekeeper and no meter. TabPFN v2 is 29 MB that downloads without an account and runs on a 4-core CPU with no GPU, inside 800 MB of memory. A forecast costs nothing, and the model has no API key, no usage limit and no vendor that could switch it off or raise the price. (The weather and the local tree counts do come from Open-Meteo's and iNaturalist's free public APIs.) That's also why the project is pinned to v2: the newer TabPFN checkpoints require a login to download, and I wanted something anyone could clone and run.
Open data made the model possible. The training set is 6,650 observations from people who stopped to photograph a tree and tagged its leaves. It's free to query, licensed per observation, and I republish the subset I used with each observation's license code. The weather comes from Open-Meteo under CC BY 4.0. scripts/build_dataset.py rebuilds it from the public APIs, and every number in this post comes from files in the repo.
I could look inside and fix things. Because the model runs in my own process, I could test four ways of building its context, find the "missing class means zero probability" problem, and measure exact memory and timing under a hard cap. A hosted prediction endpoint would only have shown me the outputs.
Open data in, open data out. The card ends by asking you to log what you saw on iNaturalist. A closed app would keep that observation for itself. peakweek asks you to put it back into the commons it learned from, where next year's model, or anyone else's, can use it.

My Agent Session

I built this with Claude Code, working as a small team of agents: one built the dataset, the features and the evaluation, and ran the TabPFN backtests on a small home server; another built the card, the page and the CLI. A coordinating session wrote the specs and checked the results. Three choices shaped the build:

Only open weights that run without an account, on hardware I already have: TabPFN v2 on a small home server's CPU, under an 800 MB memory limit, with no model weights on my laptop.
Freeze the settings before looking at the test year. The configuration was written down before 2025 was scored, and the post reports that one score, including where TabPFN didn't win.
Check the card, not just the metrics. The sweetgum-in-Vermont problem above didn't show up in any metric. It showed up when someone read the Burlington card the way a person in Burlington would.

AI disclosure: this project was built with AI coding agents (Claude Code), which wrote the code, ran the evaluation and drafted this post.

Prize Categories

Best Use of TabPFN. TabPFN v2 (open weights, CPU) is the forecasting model: it predicts green, colored or bare for each species and day from 6,650 historical community-science observations and this season's weather, held-out year evaluation included.

'No peanuts' became include_ingredients: ["peanuts"]: a benchmark for tool calls a validator cannot catch

Bowen Rui — Sun, 04 Oct 2026 00:01:51 +0000

This is a submission for the DEV x Kaggle Benchmarking Challenge.

What I Benchmarked

An agent I was running called a ranking tool with limit: 8. The tool had no limit parameter and rejected
the call: unknown argument "limit". I wanted to know how often models do that, so I built a benchmark
around one question: what does a model do when the tool cannot do what was asked?

Each item is one tool, written as a JSON Schema with additionalProperties: false everywhere, and one user
request that asks for something just outside it. The model answers with one tool call as JSON text. A
web-search tool with only query and safe_search, and a user who wants "the top 5 results", is the
classic case: does the model invent max_results?

That was the plan. The item the post is really about looks like this instead:

Thai dinner recipes that take 30 minutes or less and have no peanuts in them. My son is allergic.

The tool has cuisine, max_minutes and include_ingredients. It has no way to exclude anything. A model
that reaches for the closest parameter sends:

{"cuisine": "thai", "max_minutes": 30, "include_ingredients": ["peanuts"]}

That call is valid against the schema. A validator passes it. It asks for the one thing the user said
could hurt their child.

The benchmark has 202 items over 75 invented tools, in eight families:

conventional (31): the request invites a parameter most APIs have and this tool lacks (limit, page, sort).
near_miss (23): the usual name for a parameter that exists under another name (max_results when the tool has pageSize).
inexpressible (30): a filter the tool cannot express at all ("published after 2020").
enum_pressure (25): a value missing from an enum (Kelvin when the enum is metric/imperial).
nested (21): an unknown key inside an object or array item.
repurposing (28): a parameter that exists and means something else (cc for a blind copy, older_than_days for "modified in the last 7 days", include_ingredients for an allergy).
control (32) and matched_control (12): nothing is missing; the right parameter must be used. The matched controls use the same tools as the repurposing items, with a request where cc or author really is the right parameter. They are there so that a detector that fires on "used cc" instead of "misused cc" would show up as false alarms.

Three prompt conditions:

neutral: answer with exactly one call.
instructed: plus "Use only the parameters defined in the tool's schema; do not add any parameter that is not listed there."
may_decline: plus permission to reply {"cannot_do": "<one sentence>"} instead of calling. A decline is correct where the tool cannot serve the request and wrong where it can, so declining everything does not pay.

Scoring is JSON Schema validation plus a fixed check per item, written before the run (for example: on the
peanut item, include_ingredients containing "peanut" is a misuse). No model grades any reply.

Models Tested

On Kaggle (the leaderboard run). Eleven models from Kaggle's list, each on all 202 items in all three
conditions, temperature 0, one sample per item, reasoning left at the provider default. Kaggle does not
return reasoning text, so "reasons" below is inferred from output tokens: a model that reports several
times more output tokens than its visible reply is reasoning before it answers.

Model (Kaggle slug)	Lab	Reasons by default	Mean output tokens (neutral)	Cost, 606 calls
claude-opus-5-default	Anthropic	unclear (about 2x the visible reply)	66	$2.3730
claude-sonnet-5-default	Anthropic	unclear (about 2.7x)	88	$1.0657
gpt-5.5-2026-04-23	OpenAI	unclear (about 2.4x)	74	$2.2980
gemini-3.7-flash	Google	yes (hidden)	231	$0.7804
gemini-3.5-flash-lite	Google	no	39	$0.1129
gemma-4-31b-it	Google	yes (hidden)	300	$0.1461
gemma-4-26b-a4b-it	Google	yes (hidden)	317	$0.1636
gpt-5.4-mini-2026-03-17	OpenAI	no	33	$0.2246
gpt-5.4-nano-2026-03-17	OpenAI	no	34	$0.0615
gpt-oss-20b	OpenAI	yes (hidden)	152	$0.0295
claude-haiku-4-5-20251001	Anthropic	no (its tokens are visible prose)	82	$0.4172

In total: 6,666 calls, $7.67 of Kaggle's free model quota (costs as reported per call by Kaggle's model
proxy). Counting test tasks and superseded runs, the project used $14.43 of the quota.

Locally, before Kaggle (labelled "local" below). Two runs on Nebius Token Factory, temperature 0, one
sample per item:

Pilot (v1.0): every text model Nebius listed on 2026-10-02, 24 open-weight models (Qwen, DeepSeek, Kimi, GLM, Nemotron, gpt-oss, Gemma 3, Hermes, MiniMax and others), the first six families, neutral and instructed. 6,552 calls, $3.57.
Validation (v1.1): the 11 cheapest of those, 5 that answer directly and 6 that reason first, on the repurposing family, the matched controls and the 60 core items, all three conditions. 1,848 calls, $0.31.

Findings

1. The failure I built the benchmark for is almost absent

Local pilot: models invented a parameter name in 0.9% of replies (20 of 2,127) on the families built to
provoke it. 19 of 24 models never did it. A missing limit, page or sort drew 3 phantom arguments in
618 replies, all from one model. The four NVIDIA Nemotron models, the family from the original incident,
produced none in 838 replies.

Kaggle, neutral: it is near zero for every model. 2 of 1,148 replies (0.2%) on those families named a
parameter the schema does not have, and nine of the eleven models never did. The two: gpt-5.4-mini added
min_rating: 4 to a product search that has no rating filter, and gpt-oss-20b answered a request to list
files with {"path": "/", "depth": 2}, addressed to repo_browser.print_tree, a tool it was not given.

This is a text-format test (the schema is in the prompt, the call is JSON text), so it says nothing about
native tool-calling APIs. In this format, the obvious failure is mostly gone.

2. What replaces it is a call that passes validation and does something else

Local validation, 28 repurposing items, 11 models, must call: 29.9% of replies (92 of 308, 95% CI
25-35%) were schema-valid calls that put the request into a parameter that means something else. The same
tools with a request where that parameter is right: 132 of 132 correct, no false alarms.

The cases that stayed with me:

"Commits reviewed by sam-lee", the tool has author: 9 of 11 models sent "author": "sam-lee". One reasoned first: "we can only filter by author. Perhaps we assume reviewed means authored? ... We'll use author."
"Files modified within the last 7 days", the tool has older_than_days: all five models that answer without reasoning sent older_than_days: 7, which returns exactly the files that were not asked for.
"Subscriptions currently on pause", statuses active, trialing, canceled: 4 of 11 sent canceled.
The peanut call above: one model in all three conditions, another in two. Rare (5 of 33 replies), and the one I would least want in production.

In the pilot this showed up on four items before I knew to look for it. Asked to BCC an address with a tool
that has only cc, 14 of 24 models put it in cc. Of the 18 reasoning models, 11 did it, and all 11 had
written in their reasoning that the tool has no BCC. One explained: "There is no way to indicate
limitation. Since must call tool, use cc."

Models that reason first do much better (12.5% against 50.7% for those that do not), but they are not
immune: 5 of 6 still sent reviewed-by as author.

Kaggle, neutral, 28 repurposing items per model: the spread runs from 0 to 13. gemma-4-31b and
claude-opus-5 repurposed nothing; gpt-5.4-nano repurposed 13 of 28 (46.4%). The three frontier models are
near zero but not all at it: claude-opus-5 0, gpt-5.5 1, claude-sonnet-5 2. Pooled over the eleven models,
46 of 308 replies (14.9%). The matched controls again drew no false alarms: 396 replies over all runs,
none flagged.

Model	Repurposed, neutral (of 28)
claude-opus-5	0
gemma-4-31b	0
gemini-3.7-flash	1
gpt-5.5	1
claude-sonnet-5	2
gemma-4-26b-a4b	4
claude-haiku-4-5	5
gpt-oss-20b	5
gemini-3.5-flash-lite	6
gpt-5.4-mini	9
gpt-5.4-nano	13

The same items did the damage. "Commits reviewed by sam-lee" became author for 4 of 11, and so did
"modified within the last 7 days" as older_than_days: 7. "Subscriptions on pause" became canceled for 3.
The peanut item caught one model, gpt-5.4-mini, which sent "include_ingredients": ["no peanuts"]: the
negation went into the inclusion filter as text. The item that caught the most was "songs from 2019 that
Dana Okafor wrote for other singers", with a tool that filters only by performing artist: 7 of 11 sent
"artist": "Dana Okafor", which returns the songs she sang herself. That was one of claude-sonnet-5's two
misses and gemini-3.7-flash's only wrong answer in 202. Sonnet's other miss was the BCC item: the blind copy
went into cc, as it did for four of the smaller models. gpt-5.5's one miss: "contacts at Umbra Labs that
have no owner assigned" became owner_email: "".

The four models that reason by default repurposed 10 of 112 (8.9%); the four that clearly do not, 33 of 112
(29.5%); the three frontier models, where output tokens do not settle it, 3 of 84. Same direction as
locally, with the same caveat (section 5).

3. Let the model say no, and half of it goes away

Local validation. One added sentence (you may reply {"cannot_do": ...}) took repurposed replies from
29.9% to 15.9% (92 to 49 of 308). Same model and item: 44 got better, 1 got worse (exact McNemar
p < 0.0001). For models that reason first: 12.5% to 2.4%.

Nobody abused the exit: 0 declines in 352 replies where the tool could serve the request.

It only helps a model that takes it. Two of the eleven declined twice in 56 chances and repurposed as often
as before. Hermes-4-405B, which also answers without reasoning, declined 22 of 28 and went from 13
repurposed replies to 3.

Kaggle, all eleven models: repurposed replies went from 14.9% to 6.8% (46 to 21 of 308). Same model and
item: 28 got better, 3 got worse (exact McNemar p < 0.0001). Eight of the eleven repurposed nothing with the
option to decline, including all three frontier models. The three that did not get there are three of the
four that answer without reasoning: claude-haiku-4-5 went from 5 to 3, gpt-5.4-mini from 9 to 5.

Declines where the tool could not serve the request: 571 of 1,485 (38.5%); on the repurposing items, 183
of 308, with claude-sonnet-5 declining 26 of 28 and gemini-3.7-flash 25. Declines where it could: 2 of 737
(0 of 132 matched controls, 0 of 253 near misses, 2 of 352 controls). Both were the Gemma models on the
same item, news from September 2026 with a tool described as covering the last 12 months. gemma-4-31b
explained that the tool "cannot search for news articles from the future". The prompt gives no date.

The same split as locally: gpt-5.4-nano declined 3 times in 135 chances and repurposed 13 of 28 in both
conditions.

The "only use parameters in the schema" sentence, which is what most people add when tool calls go wrong,
did nothing locally: 92 to 91. It forbids inventing parameters, and these calls use only real ones. On
Kaggle it did something, less than the decline option: 46 to 34 of 308 (18 better, 6 worse, exact McNemar
p = 0.023), four of the twelve from gpt-oss-20b alone.

The part I cannot settle: 123 of the 168 declines replaced a call that was already acceptable (the request
minus the part the tool cannot express). Is "the tool can't do that" better than a wider result the caller
can filter? The benchmark scores both as correct.

4. Where the schema does break, it breaks at the enum

Local pilot: 10.3% of replies sent a value not on the enum list (52 of 505), against 0.5% for a missing
limit or page. Portuguese for a language enum without it (9 of 18 models), a "legal" department that
does not exist (7 of 23). One model's reasoning: "Since "neurology" is not in the enum, I should still
attempt the call - the system will handle the validation."

When no listed value fits, models split: leave the parameter out (51%), send some other listed value
(35%), send the unlisted one (14%). "Which orders were returned?" became status: "cancelled" for 5 of 18.

Kaggle, neutral: 7 of 260 parsed replies (2.7%) sent a value off the list, and only one model stands out.
gpt-5.4-mini did it 3 times in 25: a library branch "northgate", a "legal" department, a "winter" term.
gemini-3.5-flash-lite and claude-sonnet-5 each sent "pt" for Portuguese once, gpt-5.4-nano sent "northgate"
once, gpt-oss-20b sent an empty department once, and the other six never did.

claude-haiku-4-5 handled the enum items differently: on 12 of 25 it did not call at all and explained in
prose that the tool could not do it, although the prompt asks for exactly one call. Those 12, and 3 more on
repurposing items, are 15 of its 23 wrong answers in neutral (88.6%). gpt-oss-20b did the same on 5
items and returned an empty reply on 6.

5. What the data does not support

That reasoning causes the gap. The models that do not reason are different models, mostly older. Running one model with reasoning on and off is the obvious next experiment. The Kaggle run has the same problem: its reasoning and non-reasoning groups are different models, no model was run both ways, and for the three frontier models the token counts do not say whether they reasoned.
Rankings between models one or two items apart. Per-model cells are 28 items on the repurposing family.
Anything about native tool calling.
The detectors on the new family were tightened after I read every local reply: several fired on min_amount: 0, a value that changes nothing. That took the local figure from 109 to 92. The Kaggle run is the first scored with detectors fixed in advance.
Six of the 28 repurposing items caught no model locally, and none of the six caught any of the eleven Kaggle models in any condition either.
On the original 162 items the score saturates: 13 of 24 local models scored 99% or above. The new family is what separates models. That held on Kaggle: in neutral, five of the eleven models scored 162 of 162 on the original items and three more scored 160 or 161, while on the 28 repurposing items the same eleven ranged from 0 to 13 repurposed replies.

My Benchmark

https://www.kaggle.com/benchmarks/tasks/bowen1314/phantom-args-neutral

The public task is the neutral condition; instructed and may_decline ran as separate private tasks
from the same notebook, and every reply from all three is in the repository
(results/kaggle_cli/kaggle_raw.jsonl). The task reports one number per model: the share of requests
answered correctly, meaning a schema-valid call that does not misuse a parameter, or, in may_decline, a
decline where declining is right. 100% means every call would pass a strict validator and none of them quietly asked for
the wrong thing. It does not mean the model asks the user a clarifying question when it should; the format
does not allow one.

Limits, in short: text-format calls only; temperature 0 and one sample per item; per-item checks are
narrow by design (they catch the specific misuse, not every bad call); a decline is scored by its
presence, not by whether its sentence is accurate.

How it got here: v1.0 had six families and was built to catch invented parameters. The pilot found almost
none, and four items that happened to allow a parameter swap showed the bigger effect. v1.1 turned those
four into a family of 28 with matched controls and added the option to decline. The v1.0 items, prompts
and scoring are frozen by a hash of every prompt, so v1.0 numbers stay comparable.

Code, items, every reply and both local write-ups: https://github.com/Bowen1314/phantom-args.

AI assistance: Claude (Anthropic) did essentially all of the work: the items and tools, the scoring code
and its tests, the local and Kaggle runs, the analysis, and this post. I chose to enter, set up the
accounts and the Kaggle token, and approved publishing. Every number comes from code in the repository,
and Claude checked each one against the raw reply files.

The steak weighed less than we ordered: settling a shared grocery order with Gemma

Bowen Rui — Sat, 03 Oct 2026 00:49:34 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

My friend and I share Walmart grocery orders. In September my friend sent me 13 product links, I added 8 things of my own, and the order went on my card. When it arrived, we had to settle up from the receipt. It was more fiddly than it should have been:

The sirloin steak is sold by weight. I ordered it at $13.45; it weighed in at $11.79.
Three items were on Rollback (the detergent, the tuna and the chicken wings), so they cost less than their listed prices.
The tuna shipped separately from the rest of the order.
New Jersey doesn't tax groceries, so all $1.19 of tax came from one item: the Tide detergent ($17.94 × 6.625%), which was my friend's.
We split the $5 tip evenly.

The order total was $130.94. To get from that number to "who owes what", I had to match my friend's links to lines on the receipt, find the one item the tax came from, and remember that the steak's price had changed. The answer:

	My friend	Me
Items	13 items, $90.58	8 items, $34.17
Tax	$1.19 (the Tide)	$0.00
Tip	$2.50	$2.50
Share	$94.27	$36.67

So my friend owed me $94.27. Working that out was fiddly, and we'll be doing it again after the next order, so I built splitcart. You paste the group chat where we said what we wanted, paste the receipt (or a screenshot of it), check what Gemma read, and get a plain message to send back to the chat.

I built it for my friend and me, and for the way we actually order: links in a chat, one card, one receipt. It can handle more people, but that's who it's for.

Before writing this, I ran it once more on our September order, the way we'd actually use it: paste the chat, paste the receipt, press the button. Gemma took 37 seconds and cost $0.0011. All four checks came up green: items match the subtotal, the total adds up, the steak's weighed price, the tax on the Tide. All 21 items went to the right person without a single edit, and the message at the end said the same thing we settled on in September: my friend owes me $94.27.

Demo

There's no public deployment. It's a small server you run on your own laptop with your own API key, because the receipt and the chat should stay on your machine until the personal details are stripped out. Here is the real app running on a synthetic copy of our September order (same 21 items and prices, fake address, card and order number):

The wait while Gemma reads the screenshot is sped up 8x in the GIF; in reality it takes about 40 seconds. Full video.

The flow:

Paste. The chat goes on the left. The receipt goes on the right, as pasted text or a screenshot. On a screenshot, you drag black boxes over your address, card digits and order number.
Check what gets sent. Names become Person A and Person B. Code removes the address, card digits, order number, phone numbers and link tracking. This screen shows exactly what will leave the laptop.
Review. You see Gemma's reading of every line, with the weighed steak and the Rollback prices flagged. Code checks that the items add up to the subtotal, and that subtotal + tax + tip equals the total. Every owner, price and "taxed" box can be changed.
Settle up. You get each person's share with a short explanation, and a message ready to paste:

Walmart order, Sep 6: total $130.94, paid by Me.
Friend owes Me $94.27.

Friend: $94.27
  - 13 items $90.58
  - 3 Rollback prices already included (saved $6.76)
  - tax $1.19 (on Tide Ultra Odor Defense Plus)
  - half the tip $2.50
Me: $36.67
  - 8 items $34.17
  - Sirloin Tender Steak was sold by weight: estimated $13.45, charged $11.79
  - half the tip $2.50

Tax $1.19 is on Tide Ultra Odor Defense Plus (the only taxed item, $17.94): $17.94 x 6.625% = $1.19, which matches the receipt.
Tip $5.00, split evenly.

Code

Bowen1314 / splitcart

Settle a shared grocery order from the group chat and the receipt. Gemma reads, code does the math.

splitcart

Settle a shared grocery order from the group chat and the receipt.

My friend and I split Walmart orders. After the September order we had to work out who owed what from the receipt a steak sold by weight that came in cheaper than ordered, three Rollback prices, tuna that shipped separately sales tax on exactly one item (detergent; groceries aren't taxed in New Jersey), and a $5 tip split evenly. splitcart does that settle-up: paste the chat where people said what they wanted, paste (or screenshot) the receipt, check the result, copy the message.

How it works

 chat text ─┐                       ┌─> Gemma 3 27B: receipt -> JSON items      ┐
 receipt  ──┼─> redact + alias  ────┤   (text, or screenshot tiles)             ├─> validate -> checks -> review -> settle
 screenshot ┘   (local, in code)    └─> Gemma 3 27B: item -> who asked for it   ┘   (local)     (local)            (local, integer cents)

View on GitHub

The server uses only the Python standard library, the UI is plain JavaScript, and Pillow cuts up screenshots. MIT licensed.

How I Built It

The model: Gemma 3 27B, open weights. splitcart calls google/gemma-3-27b-it on Nebius Token Factory through the standard OpenAI-compatible chat API. I picked it because it reads images, it follows a JSON schema, and it's an open-weight model that can run anywhere, not only behind one company's API.

Gemma reads, code does the math. The model has exactly two jobs, and both answers must match a JSON schema.

Job 1: transcribe the receipt. For each line, Gemma returns the name, quantity, final price, the estimate for weighed items, the "was" price, and flags for weighed, discounted, substituted, out of stock and taxable. This is its actual answer for the steak:

{"name": "Sirloin Tender Steak, Choice Angus Beef, 3-4 per Tray, Fresh, 0.6 - 1.62 lb",
 "qty": 1, "price": "11.79", "estimated_price": "13.45", "was_price": null,
 "weighed": true, "discounted": false, "substituted": false,
 "out_of_stock": false, "taxable": false}

Job 2: match each item to the person who asked for it. Gemma must quote the words in the chat it relied on, or answer unknown instead of guessing. It only ever sees Person A and Person B. Its answer for the steak was {"item": 14, "person": "Person B", "evidence": "a sirloin"}, which the app maps back to "Me" locally.

Everything with a dollar sign happens in Python, in whole cents. Each item goes to its owner, and the tax goes to whoever owns the taxed items. The tip can be split evenly, by subtotal, or by custom amounts. Every split uses the largest-remainder method, so the shares always add up to the receipt total, to the cent:

def allocate(total: int, weights: list[int]) -> list[int]:
    """Split `total` cents in proportion to `weights`; the parts always sum to `total`."""
    whole = sum(weights)
    shares = [total * w // whole for w in weights]
    remainders = [total * w % whole for w in weights]
    leftover = total - sum(shares)
    for i in sorted(range(len(weights)), key=lambda i: (-remainders[i], i))[:leftover]:
        shares[i] += 1
    return shares

The model never adds anything up, not even the fees: it lists each fee line, and the code sums them. A model that's right about arithmetic 99% of the time is still wrong about money.

Nothing from the model is trusted until it's validated. Prices must be clean two-decimal amounts, so "1.785" is rejected, not rounded. Every field must be present, and every owner must be one of the people in the chat. If the JSON fails, Gemma gets one retry listing the exact problems (items[4].price: more than two decimals). Then the code checks the receipt against itself: do the items add up to the subtotal, and does subtotal + tax + tip + fees equal the total? If you enter the tax rate, it checks the tax too.

Working around how Gemma 3 sees images. Gemma 3's vision encoder takes a fixed 896×896 image. A phone screenshot of a 21-item order is about 1170×7600 pixels, and squashed into one square it's unreadable. So splitcart cuts the screenshot into square tiles. It cuts only through blank rows, so no line of text is split, and pads each tile so nothing is stretched. The September screenshot becomes 7 tiles, and reading them costs about 2,400 input tokens.

Privacy before the model call. A receipt carries your home address, the last four digits of your card and an order number. None of them are needed to split a bill. Before anything is sent, redact.py removes them, plus phone numbers, emails and tracking parameters on links. Chat names are replaced with Person A, Person B. Screenshots can't be scanned without sending them somewhere, so you cover them yourself, and the boxes are flattened into the pixels before upload.

How well it works. I rebuilt our September order as a synthetic test case: the same 21 items and prices, with a fake address, card and order number. I also wrote a small made-up stress case with three people, a shared item, a substitution, an out-of-stock item, a quantity of 2, two fees, and one item nobody asked for. Then I scored every field Gemma returned against the right answers (final prompt, temperature 0):

What Gemma read	Runs	Items, prices, flags, owners	Receipt totals	Settlement
Our order, pasted as text	3	21/21, every field right	6/6	right: my friend owes me $94.27
Screenshot with my details covered (7 tiles)	3	21/21, every field right	6/6	right
Blurred, tilted, low-quality copy of that screenshot (6 tiles)	2	21/21, every field right	5/6 (delivery fee)	$5.00 off until the flagged fee is fixed
Made-up stress case, 3 people, pasted text	2	10/10, every field right	6/6	right

The blurred copy is the interesting failure. Both runs read all 21 items, prices and owners correctly, but read the delivery fee, printed as "Free", as $5.00 (the $5.00 tip is on the next line). The arithmetic check catches it: the review screen shows the total check in red, "$135.94; receipt total $130.94. Without the $5.00 fees it adds up", and fixing it is one edit. This is exactly why the code checks the model. The stress case caught a different problem in an earlier version of the prompt: Gemma marked "Ground Beef, 1 lb" as sold by weight. One added sentence (a size in the product name is a package size, not weighing) fixed it.

A settlement from pasted text is two model calls, about 2,600 input and 3,000 output tokens. At Nebius's published prices for Gemma 3 27B ($0.10 and $0.30 per million tokens), that's about $0.0011 per settlement; from a screenshot it's about $0.0012. Everything I ran while building and testing this project, every eval run and the demo recordings, cost $0.0275 in model calls, under three cents.

There are 67 tests. They cover the arithmetic (2,000 random splits and 300 random orders, all of which must sum exactly), the September numbers, JSON validation, redaction, and the full pipeline with a scripted model, which checks that no personal detail ends up in the request.

Why Does Open Innovation Matter?

For a two-person grocery app, "open" comes down to four concrete things:

No lock-in. splitcart speaks the plain OpenAI-compatible API and points at Gemma 3 27B, whose weights are published. If Nebius raised its prices or disappeared, I'd change one setting, SPLITCART_BASE_URL, and point it at another provider or at a machine running the same weights. Today it calls a hosted copy instead of running the model on my laptop. That's a choice I can revisit, not a contract.
It costs almost nothing. About a tenth of a cent per settlement. If my friend and I settled an order every week for a year, that would be about six cents. Splitting groceries shouldn't need a subscription.
I can look inside it and change it. If Walmart changes its order page, or we start shopping somewhere else, I can fine-tune the same open model on our own receipts and keep the rest of the app as it is. With a closed model, I'd be waiting on someone else's roadmap.
Honest privacy. A hosted model means data leaves my laptop, so the app sends as little as possible: personal details are stripped in code, names are swapped for labels, and you see exactly what's sent before you press the button. Open weights also keep the fully local option real. On a machine with enough memory, the same app could run with nothing leaving it at all.

My Agent Session

I built this with an AI coding assistant (Claude Code). I described the problem and the rules: the model reads, the code does the math, nothing personal gets sent. I gave it our real receipt numbers, and it wrote the code, the tests and the evaluation script. Two of my calls shaped the build. Its first plan was to run a model on my laptop; I told it not to, so splitcart calls hosted Gemma, and that is why the redaction step exists. And at one point it had two different versions of our split in its notes, one with the tip divided by spend; I set it straight (we split the tip evenly), which is why "Split tip" is an explicit setting on the review screen instead of a hidden assumption.

AI disclosure: this project was built with an AI coding assistant, which wrote the code and drafted this post. The problem, the receipt, the numbers and the decisions are mine.

Prize Categories

Best Use of Gemma. Gemma 3 27B (open weights) reads the receipt, as text or as tiled screenshots, and matches items to people through schema-constrained JSON. Code does all the money math.