DEV Community: JinYi123456

I measured open vs closed models on my own product, and open weights won one and lost one

JinYi123456 — Sun, 18 Oct 2026 11:23:42 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Disclosure: this post and most of the code were produced with an AI coding agent working under my direction — I chose the problem, the measurements and what to keep or cut, and I checked every number by running the deployed app. The agent is not named here because the repo is deliberately a product, not a transcript. What the agent got wrong and how it was caught is in Things that only happen on a real deployment.

What I Built

A cleaner stands in a stranger's kitchen at six in the evening with the client waiting behind her. She has looked at the job for ninety seconds and now she has to name a price.

Both of her options lose money. Say "I'll email you tonight" and by tonight the client has two other quotes and picks the fastest. Quote from gut memory on the spot and the number is either padded — and lost — or short, and the job gets worked at a loss. She knows her own numbers. She just cannot do the arithmetic while somebody watches.

Chalk is the third option. She photographs the kitchen, holds a button and talks through what she would do, and by the time she is back at the van there is a priced, itemized, client-ready quote — priced in context over her own last 42 jobs, not against an industry average she does not match.

This is not a mock-up. It is the deployed build, run end to end just now on two photographs of a kitchen and an oven:

Scrub kitchen floor tiles                    1.5h   $74   [photo 1]
Clean greasy hob and splashback               2h    $112   [photo 1]
Remove and clean oven interior racks          1h     $50   [photo 2]
Degrease and sanitize oven interior          2.5h    $156   [photo 2]
Wipe down dining table and chairs            0.5h     $22   [photo 1]
Materials and consumables                            $26
                                                   ─────
Fair price                                   $318   range $254 – $394
You usually charge about $421 for move-out work this size —
this quote is $103 under your own average.

Two things happened there that a generic assistant does not do. The vision model wrote the caption on each photograph — kitchen: greasy hob, tiled splashback, oven: caked-on grime, interior racks — and attached each line item to the photograph that justifies it. And the price came with an argument: you are undercharging yourself, based on what this pro has actually billed for this kind of job.

The client never sees the win probability or the margin. They see the work, the price, the assumptions, and the photographs that explain it — on their own phone, at a link the pro texted them — with Accept and Not right now underneath.

That last part is what closes the loop. When the client taps Accept, the quote comes back as a won job: the row joins the pro's history, the win rate moves, and TabPFN refits over it. So the odds on the next quote are built from real answers rather than from the pro's memory of them.

Move	What happens	What answers
Walk	photos plus the pro's own words, typed or dictated	open-weight vision reads surfaces, sizes and price-changing conditions
Price	itemized scope → a fair price, an honest range, the odds this client says yes	TabPFN-3.5 in-context over the pro's own history — regression for price, classification for odds
Send	client message in the pro's voice, with assumptions and a validity window	an open-weight text model
Answer	one link the pro texts; the client accepts on their own phone	the same stored record, through a client-safe face

Demo

Live: https://hacktoberfestp5vqqk.tech — the real app, not a recording. (Same deployment as https://chalk-nine-flax.vercel.app, if you would rather click that.)

There is a seeded pro with 42 past jobs so the pricing model has something to reason over. Walk the first tab, add a photo or two, and press quote. Every layer reports which engine answered, so the degradation is visible too — no key for a layer, and the card says so instead of hiding it. https://hacktoberfestp5vqqk.tech/try hands a real quote to a stranger and asks one question — would you pay about $15 a month for this? — because that is the number I would rather measure than assume.

Run it offline:

git clone https://github.com/JinYi123456/chalk.git && cd chalk
npm install && npm run build:all && npm start

Film — 1:32, one unbroken take of the deployed app, with a real stopwatch on screen:

No API key is required to start it. Every layer reports the one it is missing.

Code

https://github.com/JinYi123456/chalk — MIT.

apps/web                 Vite + React PWA, plain CSS, no UI framework
services/agent           Hono + Mastra on Node
  ├─ lib/vision.ts       photo → site read            (open-weight vision)
  ├─ lib/scope.ts        site + words → itemized scope (open-weight text)
  ├─ lib/pricing.ts      TabPFN cloud → local service → transparent heuristic
  ├─ lib/benchmark.ts    the head-to-head below, with its answer key
  ├─ lib/quote-agent.ts  Mastra agent with market_rate and similar_jobs tools
  ├─ lib/memory.ts       MongoDB Atlas job memory
  ├─ lib/tiger.ts        won/lost margin time-series
  ├─ lib/temporal.ts     durable follow-ups
  └─ lib/tts.ts          ElevenLabs read-back
services/tabpfn          FastAPI, the same tabular model on local weights
services/tinker          fine-tune for the pro's quote voice
eval/                    the six walkthrough photographs and the stored run

The table I could not find

Before I wrote a line of this post I looked around at what other people were building this week, because I wanted to know what I was actually competing with. A lot of it was excellent. And in the write-ups, one sentence kept appearing:

…where your open-based approach worked better than a closed one.

That is the prompt, asking directly. And the shape of almost every answer I found was: open weights are nicer, cheaper, more private — asserted, with no measurement next to it.

So I measured it. Same walkthrough, same production prompt, sent to closed models, to hosted open-weight models, and to a model running on this laptop with no network at all. Two tasks, scored apart, so a weak pair of eyes can never be blamed on a weak brain.

The honest question is not is the model smart. It is did the answer survive the parser, because a quote is a data structure before it is a sentence. So each row is scored twice:

usable — recall over what the product could actually parse and price
in-prose — the same score read off the raw reply text before parsing

The gap between those two columns is work the model knew and still could not deliver in a form you can charge for.

Eyes — reading the walkthrough photographs

model that answered	weights	usable	in-prose	p50	tokens	$/call
`gemini-3.8-flash`	closed	4/4 · 91%	91%	10.5s	2299	$0.0021
`google/gemma-3-27b-it`	open	4/4 · 82%	82%	2.0s	840	$0.0001
`gemini-3.1-flash-lite`	closed	4/4 · 81%	81%	1.5s	2258	$0.0007
`Qwen/Qwen3-VL-30B-A3B`	open	4/4 · 79%	79%	2.5s	1754	$0.0003
`gemma3:4b` (this laptop)	open	4/4 · 74%	74%	140.3s	865	$0.0000

Open weights won this one. gemma-3-27b-it through Nebius scored 82% against the cheapest closed model's 81% — and cost seven times less. The only closed model that beat it by a real margin spent 21× the money and took five times as long to do it.

Nobody invented anything. Five arms, four photographs, zero hallucinated facts. That was a design decision, not luck: the prompt says report only what you can actually see, never invent a size you cannot gauge, and the answer key actively punishes fabrication — each case lists things that are not in the picture, and naming a carpet in a kitchen photograph is scored as an invention. Every arm cleared it. For a product where the client reads the evidence back to you, that is the number that matters most, and it is the one nobody reports.

The laptop arm is the interesting failure. 74% usable for no money at all — but two and a half minutes per read, and it got the caption count right in only three of four cases.

Brain — itemizing the scope

model that answered	weights	usable	in-prose	hours in band	p50	$/call
`gemini-3.1-flash-lite`	closed	6/6 · 82%	92%	67%	1.3s	$0.0005
`gemma-4-26b-a4b-it`	open	6/6 · 82%	90%	33%	46.5s	n/a
`Qwen3-30B-A3B` (Nebius)	open	5/6 · 73%	92%	33%	8.9s	$0.0001
`Qwen3-30B-A3B` (Featherless)	open	5/6 · 71%	87%	33%	2.5s	$0.0002
`gemini-3.8-flash`	closed	5/6 · 63%	78%	67%	8.9s	$0.0013
`gemma4:e2b` (this laptop)	open	1/6 · 11%	25%	0%	123.9s	$0.0000
`heuristicScope()`	none	6/6 · 13%	0%	67%	0.0s	$0.0000

Three findings I did not expect and do not like much:

1. Open tied closed at 82% — and the tie hides the real difference. gemma-4-26b-a4b-it and gemini-3.1-flash-lite scored identically on work found. But the open one took 46.5 seconds against 1.3, and it put the hours estimate inside a sane band on only a third of jobs against two thirds. Quality parity, decided entirely on latency and calibration. For a pro standing in a hallway, 46 seconds is not a slower answer, it is no answer.

2. The most expensive closed model came second-to-last. gemini-3.8-flash scored 63% usable — worse than a $0.0001 open model. Spending more did not buy better. It bought a bigger, slower generalist that followed the scope schema less reliably than its cheaper sibling.

3. The parser is where the work goes to die, and it is not an open-weights problem. Qwen through Nebius found 92% of the billable work in its prose and delivered 73% of it in a parseable shape. The expensive closed model lost 15 points the same way. This gap is a property of asking a language model for structured output, not of whether you can read its weights.

The laptop result is the one that stings: 25% of the work found in prose, 11% usable, one job in six parseable at all. It named the tasks. It would not put them in the boxes. So the honest version of the on-device dream, measured on my own hardware, is that a 4B model is good enough to be the camera and not good enough to be the accountant. That is a real constraint on "runs on your laptop", and I would rather publish it than quietly ship the cloud path.

And note the bottom row: the deterministic floor scored 13% on work found but 67% on hours-in-band. A coarse guess lands inside a wide band more often than a smart model finds the right work. If you only ever measure one number, you will build the wrong thing.

How to recompute all of this: node services/agent/scripts/benchmark.mjs. The answer key, the prompts and the scoring all live in services/agent/src/lib/benchmark.ts, and the prompts are imported from the same modules the product uses, so the benchmark cannot quietly drift from the thing it claims to measure. Scoring is keyword-set recall against a hand-written key — deliberately no model judging a model.

How I Built It

The open-source parts are load-bearing, not decorative.

TabPFN-3.5 (Prior Labs) is the pricing model. It does in-context learning over tables: hand it rows — job type, hours, task count, rooms, average and maximum complexity, materials, access difficulty, risk count, whether the job was won — and it predicts the next row's price without training anything on this pro's data. It answers with quantiles rather than a number, which is why a Chalk quote ships with an honest range instead of false precision. It runs three ways and says which one answered: hosted REST, a local FastAPI service on real weights, or a transparent distance-weighted heuristic over the same rows.

Mastra is the agent harness: the quote writer gets two tools, market_rate and similar_jobs, and can call them mid-sentence. MongoDB Atlas is job memory, with $vectorSearch when configured. Tiger Data holds the won/lost margin time-series. Temporal wraps the follow-up in a durable workflow so a nudge survives a restart and an accept arrives as a signal. Sentry carries one trace per compiled quote with per-stage spans. ElevenLabs reads the client message back so the pro can hear it before sending. SerpApi puts a live local rate on the card instead of a memorised one.

Tinker is the one that makes "open" mean something more than "cheap". services/tinker/train_voice.py distils the pro's own accepted client messages into context → message pairs and fine-tunes a LoRA on Qwen3-8B — four pairs, rank 16, three steps — then saves a sampler the agent calls like any other endpoint. When the sampler answers, the quote card says voice: tinker lora; when it does not, the agent's own draft stands and says whose words you are reading. It is a voice sketch, not a rewritten model, and at four pairs that is exactly what it deserves to be. What it proves is the shape: the thing that makes this pro's quotes sound like them is a file they own and can take with them, not a vendor's prompt.

Getting the answer out took one more lesson in the same category as the parser: the sampler wrote its reply into reasoning_content and left content empty, and the parser — correctly — refused it. chat_template_kwargs: {"enable_thinking": false} did not hold. Qwen3's own /no_think token did: the answer came back in content for two reasoning tokens, where widening the budget had burned two thousand and still returned nothing.

Qwen3-VL-30B-A3B and Gemma do the reading and the writing, through one OpenAI-compatible layer that walks a chain of endpoints until something answers and caches the winner — Ollama, llama.cpp, LM Studio, or any hosted open-weight catalog.

Things that only happen on a real deployment

The deployed app was broken for reasons the local app could not show. api/index.mjs exported handle(app) as its default export. Vercel reads a default export as the legacy (req, res) => void Node signature and discards the return value, so every /api request hung until the 300-second function timeout. Locally the agent serves the built PWA itself and never touches that file. The fix was to export it as the web handler it actually is — named GET and POST methods.

A latency guard meant for a human in a hallway silently disabled the product's headline feature. Prior Labs' predict call swings from ~4s to over a minute, so the hosted call gets a bounded 12-second wait and the local layer answers past that. Reasonable. Except a cold serverless instance has not fitted anything yet — that first call has to upload the history and fit before it can predict at all. So every first quote on the deployed app priced by heuristic and the pricing model never appeared. Now only the first call per instance gets the longer budget: worst case one slow quote per cold start, instead of a demo that never shows the interesting part.

Thinking models eat their own budget. Gemma 4 narrates inside a thinking block that is billed from the same max_tokens. Cap it tightly and you get an empty answer, not a short one. It gets headroom now, and the block is stripped before parsing.

A model's honest uncertainty can still be useless. The first live demo quoted $396 and printed a fair range of $53–$744. Both ends were real quantiles from a real model, over a ledger where a $70 gutter clean sits beside an $800 repaint. No client reads that as a range; they read it as a guess. The fix keeps the model's interval but clips it to the pro's own ledger — 10th to 90th percentile for that job type, never below half the recommended price. saneRange is unit-tested against that exact $53–$744 case so it cannot come back.

I asked the classifier for the win-odds ladder, and then refused to show its answer. Every other build draws the curve through one estimate — five numbers with a slope I chose. TabPFN takes a batch, so I fitted the classifier with the price charged as a feature and asked about all five rungs at once. It answered:

$219 66%  ·  $245 67%  ·  $258 67%  ·  $284 65%  ·  $315 67%

A real answer from a real fit, and useless: it tells a pro that price does not change their odds. Over 42 jobs with a 67% base rate the win/lose signal is mostly noise, so the classifier cannot see a price effect yet — and a card implying "charge $315, same chance" is worse than no card. usableBand() now requires the ladder to slope before it replaces the drawn shape, and the UI says which one you are looking at and why. When the ledger grows, the same code starts showing the model's own numbers with no change to the interface.

The writer invented its own expiry. The quote said validDays: 14 and the client message said "valid for 7 days." Two numbers, one quote — the kind of detail a pro never notices and a client holds you to. The validity window now flows from one place into the prompt, the message, the stored quote and the follow-up workflow, and a regex pass rewrites any invented window back to the real one.

And then the same class of bug bit the parser, on the demo path, at scale. I pressed quote on the deployed link while writing this and got back a three-line estimate for a walkthrough the model had itemized into six proper tasks. The card said deterministic estimate (no model reachable), which was a lie of omission — the model was reachable, had answered in 5 seconds, and had answered perfectly. The prompt tells the estimator to omit photo for a task no photograph shows. The model, being polite, wrote "photo": null. The schema wanted a number or nothing, so one null on one task failed the parse, the repair pass failed the same way, and the whole scope was thrown away.

That is finding 3 from my own table, happening to me, in production, on the flow every judge clicks first. The fix is nullish() instead of optional() — one word — plus a regression test that feeds the parser the verbatim reply and insists the six tasks survive. The client page also had a cousin of it: the header under the headline printed the expiry date as the date the quote was written, so every client was told their quote was prepared the day it stops standing.

The privacy promise was a rendering convention. Chalk's whole pitch is two faces of one record: the pro sees the odds and the margin, the client sees the work and the price. I checked what a stranger holding the client link could actually read, and the answer was everything — GET /api/jobs/<id> served winProbability, usualPrice, the market check and the hours behind every line, because the client page simply chose not to render them. A claim that only holds while nobody looks at the network tab is not a claim. The client screen now calls GET /api/jobs/<id>/client, which answers from an allow-list, and a test asserts the private fields are absent from the serialized response rather than merely invisible.

Screenshots lie, so the app prints its own split. Every quote carries the milliseconds each stage actually took, and the attribution block names the model that answered. When I could not verify the UI visually in my harness, I checked it through the accessibility tree and computed styles — which proves the handlers and the persistence but not the appearance, and I said so rather than implying I had looked.

51 hermetic specs run offline with npm test, plus the TabPFN service tests. They force offline mode, so a contributor needs no key and a CI box needs no secrets.

Why Does Open Innovation Matter?

Because the business only works open. The measured cost of the eyes on a job, at list prices: gemma-3-27b-it at $0.0001, the closed model that beats it by nine points at $0.0021. A pro who photographs four rooms on every visit, twelve jobs a day, in a busy season, is looking at a per-image bill that changes what they charge. At $0.0001 the inference is not a line item, and the whole category of tool becomes affordable to the person who needs it. That is not an aesthetic preference — it is the difference between a product a solo trade business can run and a demo that has a pricing page in its future.

Because the margin history is the family silver. What this pro charges, what they win at, and where they lose money is the most valuable thing their business owns. The whole point of pricing in-context over their own rows is that the data is the product. It should be able to live where they do: CHALK_PROVIDER=ollama moves the reading, the scoping and the writing onto the laptop, with no server that could leak it.

Because swappable is a feature, not a vibe. Every model call goes through one layer. Any OpenAI-compatible endpoint works. Nothing in the stack locks the business to a vendor's price list.

And because open has a licence edge a closed API cannot give you — and a licence trap I nearly walked into.

The edge: the standard quality scorers for this kind of evaluation are CC BY-NC-SA, which means a paid tool legally cannot use them. Apache-2.0 weights are the only path where the evaluation and the product are allowed to be the same product.

The trap, which I found while writing this and which changes how Chalk is put together: the TabPFN open weights are licensed non-commercial, and the licence text restricts using the outputs in production systems without a separate commercial licence. A "side hustle" whose pricing model runs on those weights would be a product I am not allowed to sell. So Chalk treats Prior Labs' hosted API as the commercial path — which is what the deployed demo uses — and the local weights service as the offline fallback, clearly labelled as such. The README says this out loud rather than leaving a reader to discover it after they have built a business on it.

Open here also means auditable. The answer key, the prompts and the scoring for every number above are in the repo. Nobody has to take my word for the table.

My Agent Session

The build history is in the repository, and the interesting part is not that an agent wrote most of the code but what the measurements caught that the code review did not: a deployed function that hung for 300 seconds because of an export signature, a latency guard that quietly turned off the headline feature on a cold start, and a classifier that answered a question correctly in a way that was useless to show anyone. All three were found by running the thing, not by reading it.

Prize Categories

Named, with the exact job each one does here:

Category	What it actually does in Chalk
TabPFN (Prior Labs)	The price and the win odds — both heads fitted in-context over the pro's own job history, quantiles for the range, classification for the odds, inside a wider prediction agent with tools
Mastra	The quote-writing agent itself, with `market_rate` and `similar_jobs` tools it calls mid-quote
MongoDB Atlas	Job memory: every quote stored with an embedding and recalled by similarity into the next quote
Tinker (Thinking Machines)	The quote voice: a LoRA fine-tuned on this pro's own accepted client messages (`services/tinker/train_voice.py`, four pairs, Qwen3-8B, rank 16). When it answers, the quote card says `voice: tinker lora`; when it does not, the agent's own draft stands
Tiger Data	The won/lost and margin time-series behind the profile screen — one row per quote event, queried live
Temporal	Durable follow-ups — nudge, expire, and an accept signal that survives a restart. Wired end to end in the repo; the deployed demo runs with it switched off and the health endpoint says so, which is why follow-ups there are in-process
Sentry	One trace per compiled quote with per-stage spans; the latency numbers in this post came out of those spans
SerpApi	The live local market rate on every quote — a real number, not a memorised one
ElevenLabs	Hands-free read-back of the client message, and the narration on the demo video
Gemma	The open-weight models measured head-to-head below — `gemma-4-26b-a4b-it` for scope, `google/gemma-3-27b-it` and local `gemma3:4b` for the eyes — and the on-device path (`gemma4:e2b` through Ollama) the provider chain falls back to when there is no network
a .tech domain	The address the demo runs on: https://hacktoberfestp5vqqk.tech, registered for this build and pointed at the deployment

Every one of these is wired to degrade, not to break. Pull any of them out and the quote still compiles, and the card says which layer answered.

Limits, stated plainly

Single pro. No auth, no multi-tenancy: one ledger, one device. A real product needs accounts before it needs anything else here.
The win-odds ladder is drawn, not learned, at this ledger size — and the card says so, because the classifier's own flat answer was worse than nothing.
A 4B on-device model cannot be the estimator. Measured: 11% usable. The laptop is the camera, not the accountant.
The market benchmark can be wildly off. On the run above SerpApi returned $863–$1,438 for a job Chalk priced at $318, because the source it found quotes whole-house move-out cleans. It is labelled as a live external number and it is not allowed to move the price.
A quote is not a contract. The scope is read from photographs and words; the pro still walks the job. The assumptions list exists to make that explicit to the client before they agree to anything.
Client photographs are personal data. A quote stores pictures of somebody's home, so the deletion path has to be real rather than promised: DELETE /api/jobs/<id> removes the record, its photographs and the client's link together and cancels that job's follow-up timer, and the quote view carries the button that calls it. The client link is served from a field allow-list, so the odds and the margin never cross the wire to a stranger; anyone who can reach the URL can still use it on this build, because there is no auth — the same limit as everything else here, and the reason accounts come first.
Latency is the hosted model's, not mine. Prior Labs' predict swings from ~4s to over a minute, which is why the first call per cold instance waits longer and every call after it is capped.

Chalk is named for the thing a pro always has in the van: a way to mark the job down before it stops being fresh — and then do a good job of it.

Helio: an open-weight AI that makes you touch grass — and refuses to believe you did

JinYi123456 — Sun, 11 Oct 2026 18:36:20 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 1: Touch Grass

A photo of your keyboard does not count. I checked.

What I Built

Helio is an offline-first PWA that gets you off the screen and into the world — built on open-weight models, running on your own machine. It does two jobs no other outdoors app does:

It tells you when to go. A tiny tabular foundation model — TabPFN — reads ten years of your city's weather history plus the days you personally walked, and scores the next seven days for grass-worthiness. On CPU. In about a second. No training loop — it learns in-context.
It verifies that you actually went. When you come back and photograph what you found, a local Gemma vision model checks the photo against the mission's proof criteria — on your device. No upload. If the photo is your desk, the verdict is "not grass".

Reviewing the loop visually — the dial that reads a decade of weather, the door that hands you one whisper, the pocket screen that sends you away, the stamp that only a real photo earns:

Between those two poles sits the experience:

the dial — a sun-compass that lights up the best outdoor hours of today, over a one-line weather strip (today's high/low, rain, cloud, the week ahead); one tap on use my location finds your actual sky — Helio reverse-geocodes you, keeps the nearest baked city's decade of history if you're within 50 km, and otherwise builds the climate rules from two years of open weather archive for your own coordinates
the door — a Mastra + Gemma mission compiler writes you a tiny whispered assignment ("Find the oldest standing thing within walking distance…"), and a Temporal workflow holds that mission on a durable timer until its grass window
the pocket screen — literally tells you the screen is done now
the proof — the on-device verdict, stamped with confidence and what the model saw
the journal & meadow — every verified walk gets a 5-7-5 haiku and grows a meadow with a 12-species field dex

The design rule from commit one: every dependency is a leaf node. Sever any edge of the graph and the product still works, labelled honestly — the dial shows climatology when the model is away, hand-written seeds walk in when the LLM is down, the browser reads the whisper when the voice API is missing. It survives airplane mode, backcountry, and a dead wifi router — which is exactly what an outdoors app should do. (Fun failure mode: it also survives the wrong photo. Try your keyboard.)

Demo

Live: https://helio-agent.onrender.com (free Render instance — first load may take ~1 min to wake)
Repo: https://github.com/JinYi123456/helio
79-second walkthrough — the whole loop recorded live: onboarding, helio locating you and re-reading the sky over your own coordinates, the mission compiled by Gemma, the photo verified on-device, the journal that remembers where you were standing, and the meadow grown:

https://youtu.be/ar8bz7y_dWM
90-second local run:

npm ci
ollama pull gemma4:e2b
npm run dev

On a laptop with no camera, the proof screen falls back to file upload — the app treats that as a first-class path, not an error.

The meadow only grows on verified walks — twelve species to meet, one streak to keep alive without lying:

Code

Three services, one loop:

apps/web            Vite + React PWA — dial · door · pocket · proof · journal · meadow
services/agent      Hono + Mastra — mission compiler agent, vision verifier, haiku, TTS
services/tabpfn     FastAPI — TabPFN grass-window model + precompute baker
services/tinker     Python — dataset builder + LoRA fine-tune of Helio's whisper voice

The grass window is the heart. The whole preference chain is ~20 lines:

# services/tabpfn/app/service.py
if precompute_only:                      # tiny host? serve the bake
    return load_precomputed(city_key)
try:
    model = get_model(city_key)          # TabPFN, in-context, on CPU
    scores = model.predict_days(df)
    served = "model"
except Exception:
    _mark_model_failed(city_key)         # 600s cooldown — never retry-storm
    try:
        pre = load_precomputed(city_key) # bake wins next
        pre["served"] = "precomputed-fallback"
        return pre
    except FileNotFoundError:
        scores = heuristic_scores(df, city_key)   # still honest
        served = "heuristic"

The mission compiler is one Mastra agent with three tools (grass window, local ground notes, field memory) and a zod'd draft schema. Every mission — model-written or fallback — carries photo-checkable proof criteria, because a mission you can't verify isn't a mission, it's a vibe.

The vision verdict comes back as strict JSON and is clamped on the way in — a model that hallucinates "confidence: 500" can't stamp anything:

// services/agent/src/lib/vision.ts
export function verdictFromRaw(raw: unknown, engine: string, ms: number): Verdict {
  // … clamp confidence to [0,1], default `seen`, mark `loose` when the
  // pass/fail flag is missing rather than trusting the model's prose.
}

Tests are the contract: 44 vitest specs across tools, verdicts, windows, seeds, location matching and the durability layer, plus 21 pytest specs for the forecast chain and the bring-your-own-sky coordinates (the model specs run against the real license-accepted model — see below). CI runs both on every push, and a daily GitHub Action re-bakes the grass windows on the real model and commits the JSON — the free-tier host just serves the bake.

How I Built It

Grounded in the real runtime, not in guesses. Every claim in this section was verified by running the thing: model versions, ports, timings, and every failure below is a real failure from the build log (kept in docs/building-log.md).

And when the walk ends, the local model writes it a 5-7-5 haiku and files it away — photo, verdict, weather, stamp, tags:

TabPFN's weights are license-gated. The first server boot tried to open a browser for a one-time license acceptance at 2 AM — and hung waiting on stdin. A server must never block on an interactive login. Fix that became a feature: TABPFN_NO_BROWSER=1, a 600-second failure cooldown, and the fallback chain above. The polite fix for tiny hosts: app/precompute.py bakes windows by hand or in CI; a daily Action refreshes them so the hosted demo always serves fresh, model-scored JSON.
Thinking models don't stop thinking. gemma4:e2b is a thinking model — without suppression it burns its whole budget reasoning and never answers. On Ollama's native API think:false fixes it; over the OpenAI-compatible endpoint you inject reasoning_effort:"none" via transformRequestBody. Verified: three clean tokens back instead of a 4,000-token monologue. It's in the code as a provider-scoped transform.
Your walks post-date the weather archive. Personal "I walked today" labels were silently matching nothing — the archive ends before today. Fix: synthesize feature rows from the walk-time weather (which the app records anyway) and fill the rest from same-day climatology. Now the model actually learns your good days.
React StrictMode raced the mission compiler. Dev double-mount meant two ~70-second Gemma missions started in parallel; one finished and rendered, the other overwrote state — same walk, two different mission titles. An in-flight ref guard fixed it. Production was never affected, but a confused user would have been — and a confused me at 3 AM definitely was.
Free instances are 512MB; torch is not. So the deployed grass-window service deliberately does not install tabpfn — it serves CI-baked windows, recomputes live forecasts when it can, and keeps heuristic scoring as the floor. Zero paid anything, honest served field on every response so you always know which layer answered.
Sentry's "one-line" Mastra integration silently did nothing. Node reads an ESM module graph before any module body runs, so init() called from app code lands after @mastra/core has already loaded — and already-loaded modules are never instrumented. The fix that made it real: a four-line, DSN-gated node --import preload, plus a constructed Mastra instance (a lone Agent emits nothing to an exporter). Verified against exactly what Render runs: every compile is now one gen_ai trace with token usage — and with no DSN, Sentry never enters the process at all.
A mission is a promise with a deadline — that's a workflow, not a cron. "Walk when the window opens, prove it before it closes" is exactly the shape Temporal exists for, so every compiled mission starts a missionLifecycle workflow: sleep until the TabPFN-picked golden hour, a 6-hour expiry, and one markDone signal the verifier fires when your photo passes. The workflow races sleep() against condition(done) and honestly returns "verified" or "missed". Verified for real: a mission that signalled during its sleep completed as Result: "verified" (2m33s wall time, timer cancelled), and a probe workflow woke at its golden hour, POSTed a ripe event back into the agent, then closed as "missed" — all visible in the Temporal UI. With no TEMPORAL_ADDRESS, the mission still compiles and the API says lifecycle: { scheduled: false } — leaf node, as promised.

What It Can't Do (Yet)

Honest limits, because the meadow only grows on verified walks:

It can't prove you took the photo. The vision check verifies content, not authorship — no location attestation, no sensor checks. A determined cheater with a stock photo of moss could still fool it. Helio is a nudge with a memory, not a cop.
The hosted demo answers for any coordinates, but only three skies are pre-baked. Use my location reverse-geocodes where you are: within 50 km of a baked city you get that city's sky with an honest distance note; anywhere else Helio rounds your coordinates to three decimals and keys the whole pipeline off them — a two-year climate archive fetched on demand, percentile-derived label rules, and a live forecast whenever the free forecast endpoint will answer a datacenter IP (archived climatology for your exact coordinates when it won't). With Torch on your machine that sky scores on the real model; on the free-tier host it falls back to the heuristic chain, and the served field always says which layer answered.
The deployed window service serves precomputed scores. Torch doesn't fit a 512MB free instance, so the real model runs in the daily GitHub Action bake and the service recomputes live forecasts on top of it.
On-device verdicts take ~45 seconds on CPU. That's the honest price of the photo never leaving your machine. The hosted demo uses cloud Gemma instead; locally, put the kettle on while it judges your grass.
No accounts, no cross-device sync unless you point it at MongoDB Atlas; photos and logs live in your browser's IndexedDB, exportable and burnable by you.
The hosted demo doesn't run a Temporal cluster. Durable lifecycle timers need a Temporal server next to the worker (temporal server start-dev + npm run temporal:worker); on the hosted build /api/health reports temporal: false and missions are simply not scheduled — everything else works.

Why Does Open Innovation Matter?

Every app that "helps you go outside" should itself be able to survive outside. Closed, server-dependent AI fails exactly when you need it most — on a mountain, on a plane, at the edge of a network. Helio inverts the default: the model runs on your machine, your photos never leave the device, and the whole thing costs nothing to run because there's nothing to meter.

There's also a philosophical bit I care about: an app that gamifies touching grass shouldn't demand that you stare at it. Helio's pocket screen literally sends you away. Its memory of your walks lives in your browser's IndexedDB, exportable and burnable by you. The weights are open — ollama today, llama.cpp tomorrow, the model belongs to everyone. When the product's pitch is "own your attention", the stack should practice what it preaches. That's what open weights make possible: a tool that can be entirely yours.

My Agent Session

I built Helio in an agentic coding session, and the most useful pattern was treating the agent as an accountable colleague rather than an autocomplete:

Research before code. The session started with the partner stack: what TabPFN actually is (in-context tabular foundation model — perfect for weather+habit scoring), which ports were free, which model names were real. Several "obvious" choices died in research rather than in code.
Everything ran for real. No mocked model calls. The full loop was walked in a browser against real services — the screenshots in this post are that session, including a real sprout photo stamped 95% sure, checked in 42s on CPU — and a keyboard photo was correctly rejected on an earlier pass.
Errors became features. The three nastiest bugs (license-gated weights, thinking token burn, StrictMode race) each ended up shaping the architecture — the cooldown, the provider-scoped transform, the in-flight guard. That's the log in docs/building-log.md, kept unedited.
Tests as the contract. Before touching the vision normalizer or the fallback chain, specs existed for the ugly cases: null garbage, confidence clamping, missing flags, single-hour windows. The agent's output could then be trusted precisely where trust was earned.

The honest version: the agent was brilliant at breadth (research, scaffolding, tests) and needed supervision at depth (the 2 AM browser-hang was my reviewer instinct catching a hang the code was happy to wait out). The best results came from the loop — build, verify on the real runtime, and let the failures teach.

Prize Categories

All 16 partner categories, with receipts where Helio truly uses the tech (every one of them degrades gracefully when absent — try removing any):

Category	In Helio
Render	`render.yaml` blueprint: agent (Node, serves the PWA) + grass-window service (Python, precompute mode). Free tier, two services, one blueprint.
TabPFN	The grass window itself — in-context over 10y of weather (or a 2-year archive fetched on demand for coordinates you bring), personal labels from real walks, CPU inference, license-aware fallback chain.
Gemma	Two jobs: the mission compiler (via Mastra) and the photo proof verifier — both on-device via Ollama.
Mastra	The mission compiler agent: three tools, multi-step, structured draft + repair pass.
ElevenLabs	Whisper TTS for missions, disk-cached; falls back to browser speech.
SerpApi	"What's in season right now" ground notes folded into missions.
MongoDB Atlas	Field memory over walk notes — `$vectorSearch` when configured, RAM otherwise.
Sentry	Real agent tracing: a DSN-gated `--import` preload wires runtime instrumentation, so every mission compile lands in Sentry as one trace — `http.server → mission.compile → gen_ai.invoke_agent → gen_ai.chat → http.client` — with model name, input messages and token usage as `gen_ai.*` attributes. API errors are captured too.
Tinker	Helio's whisper voice as a trainable artifact: `services/tinker/` distills the hand-written field corpus into context→whisper pairs and fine-tunes a small open-weight model over Tinker's SDK; the agent then samples whispers through Tinker's OpenAI-compatible endpoint. Wired end-to-end and env-gated — the training run is waiting on the account's billing/credit activation.
Temporal	The durable mission lifecycle: every mission is a `missionLifecycle` workflow — timer to the TabPFN-picked golden hour, 6-hour expiry, `markDone` signal from the verifier. Verified locally end-to-end: signal → `Result: "verified"`; timer → `ripe` + `missed` events posted back to the agent. No `TEMPORAL_ADDRESS`? The mission compiles anyway, honestly unscheduled.
Tiger Data	The time-series shadow of the meadow: walks land in a `footprints` hypertable (TimescaleDB, via `pg`) and `GET /api/footprints` returns the server-side 7-day series. Wired end-to-end and env-gated: no `TIGER_URL` → the endpoint answers `{enabled:false}` and the app is untouched; with a URL it's one key away.
DigitalOcean	Not used this week — Render carries the hosted demo.
Backboard	Not used — Helio's memory layer must run offline-first; a hosted control plane is against the grain.
Entire	Not used.
GitHub Copilot	Not used.
Arduino	Not used — no hardware in this build… yet. A sun-dial lamp that glows when your window opens is the obvious next step.

Helio: proof of grass. MIT-licensed, photos never leave your device, and the meadow only
grows when you actually go. See you outside.