This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle.
devchallenge #hf26challenge #gemma
What I built
A bookkeeper pastes a supplier line such as PARLE G 56G 12PK @ 48.00 INR. Holdout asks Gemma for six fields: brand, name, size, pack, unit price, and currency.
The paid part is the second measurement. A new prompt can look better on the lines used to write it. Holdout freezes that prompt, then scores it on 8 lines that were kept out of the fit. Best-of-N and a sequential fix get the same 3 model calls on each of those lines. The report prints the loss when the rewrite is worse.
The lines in the repository are synthetic. They are labeled that way. I am not describing a shop I visited.
Demo
https://holdout-zeta.vercel.app
The page is the saved run. The model is gemma4:e4b, on this laptop, through Ollama. The machine has no NVIDIA GPU. The calls went to 127.0.0.1.
| Method | Held-out field accuracy | Held-out calls |
|---|---|---|
| best of N | 0.875 | 24 |
| refine | 0.8958 | 24 |
| evolved | 0.5417 | 24 |
The evolved prompt lost to best-of-N. The fit used 17 other calls. Those calls are stored on the page and are not added to the evolved row.
On KITKAT 37G 24PK @ 720 INR, best-of-N returned a unit price of 720. The evolved prompt returned 30. Refine returned 720 and filled the product name. The page shows that row, plus Horlicks and Pepsi.
The paste box does not send the line to a server. A live run uses Ollama on the same computer. A reader who already has a Gemma server can set HOLDOUT_BASE_URL, HOLDOUT_API_KEY, and HOLDOUT_MODEL in .env. The key is not committed, and metrics.json stores the model name, not the server.
Code
https://github.com/BalkrishnaMehta/holdout
BalkrishnaMehta
/
holdout
Held-out score for a local Gemma supplier-line harness
Holdout
Holdout fits a local Gemma harness to messy supplier lines, then scores that harness on lines it did not use for fitting. Best-of-N and sequential refine get the same number of model calls on the held-out lines. If the evolved harness loses, the report says so.
The lines in data/ are synthetic. By default the model is Gemma through Ollama on this computer. A hosted OpenAI-compatible Gemma is optional: put the base URL, model name, and key in .env (see .env.example). The report names the model that ran. It does not store the server or the key.
Install
conda create -n holdout python=3.11 pip -y
conda activate holdout
pip install -e ".[dev]"
ollama pull gemma4:e4b
gemma3:4b is used only when E4B cannot return one JSON object and .env does not name another model.
copy .env.example .env
Run
$env:OLLAMA_HOST=…The license is MIT. data/fit.jsonl has 16 lines. data/heldout.jsonl has 8. No fit id appears in the held-out file.
How I built it
Wang et al. compare a searched prompt with extra samples at a matched call budget, and they score the result on lines the search did not use. The paper is arXiv 2607.12227.
Holdout keeps that comparison small enough for a CPU. One rewrite. The fit pass counts missed fields with the labels. One more call asks for four short rules. After the text is generated, the same labels score the fields. They are not copied into the extraction prompt, and they are not used to choose a held-out line. pick_first_valid keeps the first complete JSON object. It does not look at the label.
The rewrite from this run talked about people's names, titles such as Mr. and Dr., and currency written as LaTeX. It had been asked for rules about name, currency, and brand, which were the fields missed most often on the fit lines. Held-out field accuracy went from 0.875 to 0.5417. I left that loss in data/metrics.json. I did not edit the rules afterward to produce a win.
gemma3:4b is installed for the case where gemma4:e4b cannot return one JSON object. This run returned JSON from E4B, so the fallback stayed unused.
Two DEV posts sat next to this design. Ollama Structured Outputs in Practice shows a typed JSON parse from a local model. Variance testing flipped my Ollama benchmark ranking shows a single pass changing which local model looks best. Holdout still takes the first JSON object in the reply and scores the six fields by exact match. A schema helper would not answer the held-out question.
Why open innovation matters
Gemma 4 E4B is an open-weight model. This laptop has 16 GB of RAM and an Intel CPU. The 89 calls behind the table, 17 on the fit and 72 on the held-out lines, stayed on that machine.
A closed extraction API would bill each of those calls. A bookkeeper running the comparison on a stack of supplier lines would pay that bill every evening. A local model makes the check cheap enough to run on lines that should not leave the shop.
Prize category
Best Use of Gemma. The saved model name is gemma4:e4b, through Ollama. This post enters that category only.
First user
No one else has used Holdout. I have not been paid. This section stays empty until someone who is not me runs it on lines I did not write.
Top comments (0)