I measured open vs closed models on my own product, and open weights won one and lost one

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Disclosure: this post and most of the code were produced with an AI coding agent working under my direction — I chose the problem, the measurements and what to keep or cut, and I checked every number by running the deployed app. The agent is not named here because the repo is deliberately a product, not a transcript. What the agent got wrong and how it was caught is in Things that only happen on a real deployment.

What I Built

A cleaner stands in a stranger's kitchen at six in the evening with the client waiting behind her. She has looked at the job for ninety seconds and now she has to name a price.

Both of her options lose money. Say "I'll email you tonight" and by tonight the client has two other quotes and picks the fastest. Quote from gut memory on the spot and the number is either padded — and lost — or short, and the job gets worked at a loss. She knows her own numbers. She just cannot do the arithmetic while somebody watches.

Chalk is the third option. She photographs the kitchen, holds a button and talks through what she would do, and by the time she is back at the van there is a priced, itemized, client-ready quote — priced in context over her own last 42 jobs, not against an industry average she does not match.

This is not a mock-up. It is the deployed build, run end to end just now on two photographs of a kitchen and an oven:

Scrub kitchen floor tiles                    1.5h   $74   [photo 1]
Clean greasy hob and splashback               2h    $112   [photo 1]
Remove and clean oven interior racks          1h     $50   [photo 2]
Degrease and sanitize oven interior          2.5h    $156   [photo 2]
Wipe down dining table and chairs            0.5h     $22   [photo 1]
Materials and consumables                            $26
                                                   ─────
Fair price                                   $318   range $254 – $394
You usually charge about $421 for move-out work this size —
this quote is $103 under your own average.

Two things happened there that a generic assistant does not do. The vision model wrote the caption on each photograph — kitchen: greasy hob, tiled splashback, oven: caked-on grime, interior racks — and attached each line item to the photograph that justifies it. And the price came with an argument: you are undercharging yourself, based on what this pro has actually billed for this kind of job.

The client never sees the win probability or the margin. They see the work, the price, the assumptions, and the photographs that explain it — on their own phone, at a link the pro texted them — with Accept and Not right now underneath.

That last part is what closes the loop. When the client taps Accept, the quote comes back as a won job: the row joins the pro's history, the win rate moves, and TabPFN refits over it. So the odds on the next quote are built from real answers rather than from the pro's memory of them.

Move	What happens	What answers
Walk	photos plus the pro's own words, typed or dictated	open-weight vision reads surfaces, sizes and price-changing conditions
Price	itemized scope → a fair price, an honest range, the odds this client says yes	TabPFN-3.5 in-context over the pro's own history — regression for price, classification for odds
Send	client message in the pro's voice, with assumptions and a validity window	an open-weight text model
Answer	one link the pro texts; the client accepts on their own phone	the same stored record, through a client-safe face

Demo

Live: https://hacktoberfestp5vqqk.tech — the real app, not a recording. (Same deployment as https://chalk-nine-flax.vercel.app, if you would rather click that.)

There is a seeded pro with 42 past jobs so the pricing model has something to reason over. Walk the first tab, add a photo or two, and press quote. Every layer reports which engine answered, so the degradation is visible too — no key for a layer, and the card says so instead of hiding it. https://hacktoberfestp5vqqk.tech/try hands a real quote to a stranger and asks one question — would you pay about $15 a month for this? — because that is the number I would rather measure than assume.

Run it offline:

git clone https://github.com/JinYi123456/chalk.git && cd chalk
npm install && npm run build:all && npm start

Film — 1:32, one unbroken take of the deployed app, with a real stopwatch on screen:

No API key is required to start it. Every layer reports the one it is missing.

Code

https://github.com/JinYi123456/chalk — MIT.

apps/web                 Vite + React PWA, plain CSS, no UI framework
services/agent           Hono + Mastra on Node
  ├─ lib/vision.ts       photo → site read            (open-weight vision)
  ├─ lib/scope.ts        site + words → itemized scope (open-weight text)
  ├─ lib/pricing.ts      TabPFN cloud → local service → transparent heuristic
  ├─ lib/benchmark.ts    the head-to-head below, with its answer key
  ├─ lib/quote-agent.ts  Mastra agent with market_rate and similar_jobs tools
  ├─ lib/memory.ts       MongoDB Atlas job memory
  ├─ lib/tiger.ts        won/lost margin time-series
  ├─ lib/temporal.ts     durable follow-ups
  └─ lib/tts.ts          ElevenLabs read-back
services/tabpfn          FastAPI, the same tabular model on local weights
services/tinker          fine-tune for the pro's quote voice
eval/                    the six walkthrough photographs and the stored run

The table I could not find

Before I wrote a line of this post I looked around at what other people were building this week, because I wanted to know what I was actually competing with. A lot of it was excellent. And in the write-ups, one sentence kept appearing:

…where your open-based approach worked better than a closed one.

That is the prompt, asking directly. And the shape of almost every answer I found was: open weights are nicer, cheaper, more private — asserted, with no measurement next to it.

So I measured it. Same walkthrough, same production prompt, sent to closed models, to hosted open-weight models, and to a model running on this laptop with no network at all. Two tasks, scored apart, so a weak pair of eyes can never be blamed on a weak brain.

The honest question is not is the model smart. It is did the answer survive the parser, because a quote is a data structure before it is a sentence. So each row is scored twice:

• usable — recall over what the product could actually parse and price
• in-prose — the same score read off the raw reply text before parsing

The gap between those two columns is work the model knew and still could not deliver in a form you can charge for.

Eyes — reading the walkthrough photographs

model that answered	weights	usable	in-prose	invented facts	p50	tokens	$/call
gemini-3.8-flash	closed	4/4 · 91%	91%	0	10.5s	2299	$0.0021
google/gemma-3-27b-it	open	4/4 · 82%	82%	0	2.0s	840	$0.0001
gemini-3.1-flash-lite	closed	4/4 · 81%	81%	0	1.5s	2258	$0.0007
Qwen/Qwen3-VL-30B-A3B	open	4/4 · 79%	79%	0	2.5s	1754	$0.0003
gemma3:4b (this laptop)	open	4/4 · 74%	74%	0	140.3s	865	$0.0000

Open weights won this one. gemma-3-27b-it through Nebius scored 82% against the cheapest closed model's 81% — and cost seven times less. The only closed model that beat it by a real margin spent 21× the money and took five times as long to do it.

Nobody invented anything. Five arms, four photographs, zero hallucinated facts. That was a design decision, not luck: the prompt says report only what you can actually see, never invent a size you cannot gauge, and the answer key actively punishes fabrication — each case lists things that are not in the picture, and naming a carpet in a kitchen photograph is scored as an invention. Every arm cleared it. For a product where the client reads the evidence back to you, that is the number that matters most, and it is the one nobody reports.

The laptop arm is the interesting failure. 74% usable for no money at all — but two and a half minutes per read, and it got the caption count right in only three of four cases.

Brain — itemizing the scope

model that answered	weights	usable	in-prose	hours in band	p50	$/call
gemini-3.1-flash-lite	closed	6/6 · 82%	92%	67%	1.3s	$0.0005
gemma-4-26b-a4b-it	open	6/6 · 82%	90%	33%	46.5s	n/a
Qwen3-30B-A3B (Nebius)	open	5/6 · 73%	92%	33%	8.9s	$0.0001
Qwen3-30B-A3B (Featherless)	open	5/6 · 71%	87%	33%	2.5s	$0.0002
gemini-3.8-flash	closed	5/6 · 63%	78%	67%	8.9s	$0.0013
gemma4:e2b (this laptop)	open	1/6 · 11%	25%	0%	123.9s	$0.0000
heuristicScope()	none	6/6 · 13%	0%	67%	0.0s	$0.0000

Three findings I did not expect and do not like much:

1. Open tied closed at 82% — and the tie hides the real difference. gemma-4-26b-a4b-it and gemini-3.1-flash-lite scored identically on work found. But the open one took 46.5 seconds against 1.3, and it put the hours estimate inside a sane band on only a third of jobs against two thirds. Quality parity, decided entirely on latency and calibration. For a pro standing in a hallway, 46 seconds is not a slower answer, it is no answer.

2. The most expensive closed model came second-to-last. gemini-3.8-flash scored 63% usable — worse than a $0.0001 open model. Spending more did not buy better. It bought a bigger, slower generalist that followed the scope schema less reliably than its cheaper sibling.

3. The parser is where the work goes to die, and it is not an open-weights problem. Qwen through Nebius found 92% of the billable work in its prose and delivered 73% of it in a parseable shape. The expensive closed model lost 15 points the same way. This gap is a property of asking a language model for structured output, not of whether you can read its weights.

The laptop result is the one that stings: 25% of the work found in prose, 11% usable, one job in six parseable at all. It named the tasks. It would not put them in the boxes. So the honest version of the on-device dream, measured on my own hardware, is that a 4B model is good enough to be the camera and not good enough to be the accountant. That is a real constraint on "runs on your laptop", and I would rather publish it than quietly ship the cloud path.

And note the bottom row: the deterministic floor scored 13% on work found but 67% on hours-in-band. A coarse guess lands inside a wide band more often than a smart model finds the right work. If you only ever measure one number, you will build the wrong thing.

How to recompute all of this: node services/agent/scripts/benchmark.mjs. The answer key, the prompts and the scoring all live in services/agent/src/lib/benchmark.ts, and the prompts are imported from the same modules the product uses, so the benchmark cannot quietly drift from the thing it claims to measure. Scoring is keyword-set recall against a hand-written key — deliberately no model judging a model.

How I Built It

The open-source parts are load-bearing, not decorative.

TabPFN-3.5 (Prior Labs) is the pricing model. It does in-context learning over tables: hand it rows — job type, hours, task count, rooms, average and maximum complexity, materials, access difficulty, risk count, whether the job was won — and it predicts the next row's price without training anything on this pro's data. It answers with quantiles rather than a number, which is why a Chalk quote ships with an honest range instead of false precision. It runs three ways and says which one answered: hosted REST, a local FastAPI service on real weights, or a transparent distance-weighted heuristic over the same rows.

Mastra is the agent harness: the quote writer gets two tools, market_rate and similar_jobs, and can call them mid-sentence. MongoDB Atlas is job memory, with $vectorSearch when configured. Tiger Data holds the won/lost margin time-series. Temporal wraps the follow-up in a durable workflow so a nudge survives a restart and an accept arrives as a signal. Sentry carries one trace per compiled quote with per-stage spans. ElevenLabs reads the client message back so the pro can hear it before sending. SerpApi puts a live local rate on the card instead of a memorised one.

Tinker is the one that makes "open" mean something more than "cheap". services/tinker/train_voice.py distils the pro's own accepted client messages into context → message pairs and fine-tunes a LoRA on Qwen3-8B — four pairs, rank 16, three steps — then saves a sampler the agent calls like any other endpoint. When the sampler answers, the quote card says voice: tinker lora; when it does not, the agent's own draft stands and says whose words you are reading. It is a voice sketch, not a rewritten model, and at four pairs that is exactly what it deserves to be. What it proves is the shape: the thing that makes this pro's quotes sound like them is a file they own and can take with them, not a vendor's prompt.

Getting the answer out took one more lesson in the same category as the parser: the sampler wrote its reply into reasoning_content and left content empty, and the parser — correctly — refused it. chat_template_kwargs: {"enable_thinking": false} did not hold. Qwen3's own /no_think token did: the answer came back in content for two reasoning tokens, where widening the budget had burned two thousand and still returned nothing.

Qwen3-VL-30B-A3B and Gemma do the reading and the writing, through one OpenAI-compatible layer that walks a chain of endpoints until something answers and caches the winner — Ollama, llama.cpp, LM Studio, or any hosted open-weight catalog.

Things that only happen on a real deployment

The deployed app was broken for reasons the local app could not show. api/index.mjs exported handle(app) as its default export. Vercel reads a default export as the legacy (req, res) => void Node signature and discards the return value, so every /api request hung until the 300-second function timeout. Locally the agent serves the built PWA itself and never touches that file. The fix was to export it as the web handler it actually is — named GET and POST methods.

A latency guard meant for a human in a hallway silently disabled the product's headline feature. Prior Labs' predict call swings from ~4s to over a minute, so the hosted call gets a bounded 12-second wait and the local layer answers past that. Reasonable. Except a cold serverless instance has not fitted anything yet — that first call has to upload the history and fit before it can predict at all. So every first quote on the deployed app priced by heuristic and the pricing model never appeared. Now only the first call per instance gets the longer budget: worst case one slow quote per cold start, instead of a demo that never shows the interesting part.

Thinking models eat their own budget. Gemma 4 narrates inside a thinking block that is billed from the same max_tokens. Cap it tightly and you get an empty answer, not a short one. It gets headroom now, and the block is stripped before parsing.

A model's honest uncertainty can still be useless. The first live demo quoted $396 and printed a fair range of $53–$744. Both ends were real quantiles from a real model, over a ledger where a $70 gutter clean sits beside an $800 repaint. No client reads that as a range; they read it as a guess. The fix keeps the model's interval but clips it to the pro's own ledger — 10th to 90th percentile for that job type, never below half the recommended price. saneRange is unit-tested against that exact $53–$744 case so it cannot come back.

I asked the classifier for the win-odds ladder, and then refused to show its answer. Every other build draws the curve through one estimate — five numbers with a slope I chose. TabPFN takes a batch, so I fitted the classifier with the price charged as a feature and asked about all five rungs at once. It answered:

$219 66%  ·  $245 67%  ·  $258 67%  ·  $284 65%  ·  $315 67%

A real answer from a real fit, and useless: it tells a pro that price does not change their odds. Over 42 jobs with a 67% base rate the win/lose signal is mostly noise, so the classifier cannot see a price effect yet — and a card implying "charge $315, same chance" is worse than no card. usableBand() now requires the ladder to slope before it replaces the drawn shape, and the UI says which one you are looking at and why. When the ledger grows, the same code starts showing the model's own numbers with no change to the interface.

The writer invented its own expiry. The quote said validDays: 14 and the client message said "valid for 7 days." Two numbers, one quote — the kind of detail a pro never notices and a client holds you to. The validity window now flows from one place into the prompt, the message, the stored quote and the follow-up workflow, and a regex pass rewrites any invented window back to the real one.

And then the same class of bug bit the parser, on the demo path, at scale. I pressed quote on the deployed link while writing this and got back a three-line estimate for a walkthrough the model had itemized into six proper tasks. The card said deterministic estimate (no model reachable), which was a lie of omission — the model was reachable, had answered in 5 seconds, and had answered perfectly. The prompt tells the estimator to omit photo for a task no photograph shows. The model, being polite, wrote "photo": null. The schema wanted a number or nothing, so one null on one task failed the parse, the repair pass failed the same way, and the whole scope was thrown away.

That is finding 3 from my own table, happening to me, in production, on the flow every judge clicks first. The fix is nullish() instead of optional() — one word — plus a regression test that feeds the parser the verbatim reply and insists the six tasks survive. The client page also had a cousin of it: the header under the headline printed the expiry date as the date the quote was written, so every client was told their quote was prepared the day it stops standing.

The privacy promise was a rendering convention. Chalk's whole pitch is two faces of one record: the pro sees the odds and the margin, the client sees the work and the price. I checked what a stranger holding the client link could actually read, and the answer was everything — GET /api/jobs/<id> served winProbability, usualPrice, the market check and the hours behind every line, because the client page simply chose not to render them. A claim that only holds while nobody looks at the network tab is not a claim. The client screen now calls GET /api/jobs/<id>/client, which answers from an allow-list, and a test asserts the private fields are absent from the serialized response rather than merely invisible.

Screenshots lie, so the app prints its own split. Every quote carries the milliseconds each stage actually took, and the attribution block names the model that answered. When I could not verify the UI visually in my harness, I checked it through the accessibility tree and computed styles — which proves the handlers and the persistence but not the appearance, and I said so rather than implying I had looked.

51 hermetic specs run offline with npm test, plus the TabPFN service tests. They force offline mode, so a contributor needs no key and a CI box needs no secrets.

Why Does Open Innovation Matter?

Because the business only works open. The measured cost of the eyes on a job, at list prices: gemma-3-27b-it at $0.0001, the closed model that beats it by nine points at $0.0021. A pro who photographs four rooms on every visit, twelve jobs a day, in a busy season, is looking at a per-image bill that changes what they charge. At $0.0001 the inference is not a line item, and the whole category of tool becomes affordable to the person who needs it. That is not an aesthetic preference — it is the difference between a product a solo trade business can run and a demo that has a pricing page in its future.

Because the margin history is the family silver. What this pro charges, what they win at, and where they lose money is the most valuable thing their business owns. The whole point of pricing in-context over their own rows is that the data is the product. It should be able to live where they do: CHALK_PROVIDER=ollama moves the reading, the scoping and the writing onto the laptop, with no server that could leak it.

Because swappable is a feature, not a vibe. Every model call goes through one layer. Any OpenAI-compatible endpoint works. Nothing in the stack locks the business to a vendor's price list.

And because open has a licence edge a closed API cannot give you — and a licence trap I nearly walked into.

The edge: the standard quality scorers for this kind of evaluation are CC BY-NC-SA, which means a paid tool legally cannot use them. Apache-2.0 weights are the only path where the evaluation and the product are allowed to be the same product.

The trap, which I found while writing this and which changes how Chalk is put together: the TabPFN open weights are licensed non-commercial, and the licence text restricts using the outputs in production systems without a separate commercial licence. A "side hustle" whose pricing model runs on those weights would be a product I am not allowed to sell. So Chalk treats Prior Labs' hosted API as the commercial path — which is what the deployed demo uses — and the local weights service as the offline fallback, clearly labelled as such. The README says this out loud rather than leaving a reader to discover it after they have built a business on it.

Open here also means auditable. The answer key, the prompts and the scoring for every number above are in the repo. Nobody has to take my word for the table.

My Agent Session

The build history is in the repository, and the interesting part is not that an agent wrote most of the code but what the measurements caught that the code review did not: a deployed function that hung for 300 seconds because of an export signature, a latency guard that quietly turned off the headline feature on a cold start, and a classifier that answered a question correctly in a way that was useless to show anyone. All three were found by running the thing, not by reading it.

Prize Categories

Named, with the exact job each one does here:

Category	What it actually does in Chalk
TabPFN (Prior Labs)	The price and the win odds — both heads fitted in-context over the pro's own job history, quantiles for the range, classification for the odds, inside a wider prediction agent with tools
Mastra	The quote-writing agent itself, with market_rate and similar_jobs tools it calls mid-quote
MongoDB Atlas	Job memory: every quote stored with an embedding and recalled by similarity into the next quote
Tinker (Thinking Machines)	The quote voice: a LoRA fine-tuned on this pro's own accepted client messages (services/tinker/train_voice.py, four pairs, Qwen3-8B, rank 16). When it answers, the quote card says voice: tinker lora; when it does not, the agent's own draft stands
Tiger Data	The won/lost and margin time-series behind the profile screen — one row per quote event, queried live
Temporal	Durable follow-ups — nudge, expire, and an accept signal that survives a restart. Wired end to end in the repo; the deployed demo runs with it switched off and the health endpoint says so, which is why follow-ups there are in-process
Sentry	One trace per compiled quote with per-stage spans; the latency numbers in this post came out of those spans
SerpApi	The live local market rate on every quote — a real number, not a memorised one
ElevenLabs	Hands-free read-back of the client message, and the narration on the demo video
Gemma	The open-weight models measured head-to-head below — gemma-4-26b-a4b-it for scope, google/gemma-3-27b-it and local gemma3:4b for the eyes — and the on-device path (gemma4:e2b through Ollama) the provider chain falls back to when there is no network
a .tech domain	The address the demo runs on: https://hacktoberfestp5vqqk.tech, registered for this build and pointed at the deployment

Every one of these is wired to degrade, not to break. Pull any of them out and the quote still compiles, and the card says which layer answered.

Limits, stated plainly

• Single pro. No auth, no multi-tenancy: one ledger, one device. A real product needs accounts before it needs anything else here.
• The win-odds ladder is drawn, not learned, at this ledger size — and the card says so, because the classifier's own flat answer was worse than nothing.
• A 4B on-device model cannot be the estimator. Measured: 11% usable. The laptop is the camera, not the accountant.
• The market benchmark can be wildly off. On the run above SerpApi returned $863–$1,438 for a job Chalk priced at $318, because the source it found quotes whole-house move-out cleans. It is labelled as a live external number and it is not allowed to move the price.
• A quote is not a contract. The scope is read from photographs and words; the pro still walks the job. The assumptions list exists to make that explicit to the client before they agree to anything.
• Client photographs are personal data. A quote stores pictures of somebody's home, so the deletion path has to be real rather than promised: DELETE /api/jobs/<id> removes the record, its photographs and the client's link together and cancels that job's follow-up timer, and the quote view carries the button that calls it. The client link is served from a field allow-list, so the odds and the margin never cross the wire to a stranger; anyone who can reach the URL can still use it on this build, because there is no auth — the same limit as everything else here, and the reason accounts come first.
• Latency is the hosted model's, not mine. Prior Labs' predict swings from ~4s to over a minute, which is why the first call per cold instance waits longer and every call after it is capped.

Chalk is named for the thing a pro always has in the van: a way to mark the job down before it stops being fresh — and then do a good job of it.