DEV Community: Sankalp Kotewar

The AI didn't invent my client update. It promoted it.

Sankalp Kotewar — Wed, 21 Oct 2026 05:42:49 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

What I Built

Every Friday I owe a client a paragraph about what happened that week. The information is all sitting in git log. What I do not have, at 6pm on a Friday, is the will to reconstruct it from forty commit messages and a diff I have already forgotten writing.

Receipts reads a week of commits with a local model and writes the update. Then it does the part I actually care about: it checks every sentence against the commit it cites, and tells me which sentences the commit does not support.

The thing I found is that the model almost never invents work. It promotes it. Pin requests<3 becomes "refactored to pin requests to version 2.31.0" — a commit that removes 2.31.0. Run black becomes "implemented the export function" — black reformatted it, a different commit wrote it.

Every one of those sentences is locally defensible. Collectively they are a lie, and nobody checks a status report. That is the exact seam this project sits in.

Demo

kotewar.github.io/receipts

The page opens as the report a client would receive: six confident bullets, no citations, nothing to click. Flip the switch and every sentence grows the commit it came from — the sha, the message, the files, the diff, and a verdict.

Three of the six receipts do not support the sentence above them. The report reads perfectly either way. That is the whole project in one interaction.

The demo is a saved run baked into a static page, because the model is 3.4GB and GitHub Pages does not have a CPU. What is replayed is a real run, unedited: data/demo.json is the exact model output, and the verdicts are the output of the real checker, not a mock-up. The live model runs from the CLI:

PYTHONPATH=. .venv/bin/python scripts/report.py ~/code/some-client --limit 40

Pointed at the repo for my previous project — real code, real commits, not a fixture:

  parchi: 9 commits, 2026-10-17 -> 2026-10-17

  ok   The counselling text has been translated into a native Hindi voice using
       ElevenLabs, resulting in a 39.0-second audio file with 610 KB and a
       44.1 kHz sample rate
  FAIL The LICENSE file has been added, clarifying that this project is licensed
       under the MIT License
       [715e025] does not support — Install TabPFN for real, and pin NumPy under 2
       needed [36dd958] Add an MIT license, and say plainly what this is not
  FAIL The `README.md` file has been updated to include a self-check script
       (`scripts/selfcheck.py`) that verifies the functionality of the system
       [ced916a] does not support — Stop the counselling saying the strength twice

  2 of 6 receipts do not support their sentence.

The second failure is my favourite in the whole project. ced916a is the commit that created scripts/selfcheck.py. The README line documenting it came in the commit just before, 744075a. The model found the right file and the wrong commit — which is precisely the error a human skims past, because the file exists and the sentence is true.

Code

github.com/kotewar/receipts

app/trace.py — the three deterministic rules. No model, no API call, no network.
app/receipts.py — finds the commit that would have supported a bad sentence, when one exists
scripts/eval.py — the harness behind every number below
scripts/report.py — the CLI above
scripts/make_fixture_repo.py — builds the 12-commit demo repo, including its traps
data/eval_base.json, data/eval_primed.json — both runs, committed so the claims are checkable

How I Built It

Start with the failure you can actually catch

I did not want to build a plagiarism detector or a "hallucination score". I wanted to catch one specific thing: a sentence whose receipt does not support it. So the gate has three rules, all boring, all deterministic.

Grounding. Numbers and identifiers in the sentence must appear in the cited commit. Including the distinction that broke a naive version: a version that appears only in deleted lines was removed, not introduced. Without that, "pinned requests to 2.31.0" passes against a commit that changed requests==2.31.0 to requests<3, because 2.31.0 is right there in the diff. A claim can be exactly backwards and satisfy a string search.

Inflation. The sentence's leading verb is ranked against the commit message's leading verb. refactored on a Pin commit is two ranks apart. The claim is not that one verb is worse than another; it is that the sentence and its receipt are describing different sizes of work.

Scope. "the system", "the platform", "the codebase" on a commit that touched one file.

That is the whole gate. It is a triage tool, not a judge. It does not decide whether a sentence is good. It decides which sentences I have to read.

The traps are the test

The eval repo is twelve synthetic commits from one week of a small invoicing client. Two of them touch hundreds of lines and mean nothing (Run black, Rename qty to quantity). One touches six lines and matters (Fix null check in parse_invoice). If a checker ranks by churn, it will point at the wrong commit every time — which is exactly what the model does.

What it caught

Five real outputs from the saved runs. Left is what the model cited; right is what it said.

cited	the update said
`Pin requests<3`	"Refactored the code to pin `requests` to version 2.31.0"
`Run black`	"Implemented an `export_rows` function to generate a CSV file"
`Add retry to fetch_rate`	"Fixed a null check in `parse_invoice`"
`Document the retry behaviour`	"Implemented retry logic for fetching rates"
`Pin requests<3`	"Added a new test case (`test_empty.py`)"

Every one is a real commit, so a link-checker passes all five. Four of the five are misattribution, not fabrication: the work happened, the receipt is wrong. The fifth — a README commit claimed as an implementation — is the one that stings, because documentation commits are the easiest thing in a repo to claim as engineering.

What it measures

Two prompt variants, three runs each, six bullets per run: 36 bullets.

13 cited a commit that does not support them. The gate flagged all 13.
4 more were wrong in a way the gate cannot see.

That second number is the interesting one, and it only came from reading the sentences the gate approved. A commit that changed a sleep constant from 0.2 to 0.25 produced this:

"Improved error handling within rates.fetch_rate by introducing a retry mechanism with exponential backoff"

The retry mechanism was introduced two commits earlier. This commit tuned its delay. The verbs agree, there is no number to ground, and no filename is named, so all three rules pass it. A gate that only reports its own flags cannot find its own misses. The honest total is 17 of 36, with 13 caught.

The same class also produced the best sentence in the dataset, from the primed variant:

"Improved error handling by adjusting the sleep time within rates.fetch_rate to 0.25 * (attempt + 1)"

Same commit, same model, and it named the constant exactly. Priming the prompt ("be literal, do not upgrade a verb") did not reliably reduce the flag rate — 7/18 versus 6/18 is noise — but it did move the surviving sentences toward the literal. That is a real but modest win, and I am not going to dress it up as more.

Three bugs the output found, not the tests

Each of these was caught by reading a flagged line and thinking "that sentence is fine".

A filename read as a verb. "Ran black to format the code in export.py, invoice.py, and rates.py" was flagged for claiming export at strength 3. It was the filename. Correct sentence, wrong flag.

Subordinate clauses inflating the verb. "Bumped the version to 1.2.1, reflecting the updated retry logic" was flagged because retry ranks higher than bump. The claim is the main clause; the tail was context.

Commit messages scored by their most impressive word. This one was hiding a false negative. "Document the retry behaviour" scored as strong as "Add retry to fetch_rate", because retry is also a verb, so "implemented retry logic" cited to a docs commit passed clean. Commit messages lead with their verb. Read the first one, on both sides.

All three are fixed, and all three are now cases in the checker's own test set. None of them came from a test I wrote in advance.

The Lazy Payoff

The theme this week is laziness, so here is what actually got removed from my Friday.

I no longer open the repo and reconstruct the week from memory. I run one command, wait about two minutes on an eight-core laptop with no GPU, and get six bullets. That is the obvious saving and it is the smaller half.

The bigger half is that I do not proofread the whole report. I read the flagged sentences and the receipts behind them. On this run that is three sentences instead of six, each with the diff already attached, and one of them already tells me which commit I meant. The work I stopped doing is the part I was worst at: checking a plausible sentence against a commit I half-remember.

There is a second-order laziness that I did not expect and would not give up. Because the model runs locally for nothing, I could run the eval again and again, read the raw output, and rewrite the prompt without thinking about a bill. Every bug in the previous section was found that way. That is not a workflow you get when each run is a metered API call.

Why Does Open Innovation Matter?

The diffs are the client's code. This tool reads full diffs, not just commit messages — that is what makes the receipts meaningful. Sending a client's private repository to a hosted API every Friday is a data-processing decision I would have to explain to the client. Running the model on the laptop the repo is already on makes it a non-decision. The data does not move.

The checker is model-independent, which is only useful if the model is swappable. app/trace.py checks the sentence, not the model. That means the sentence generator is a replaceable part: gemma3:4b today, something smaller or fine-tuned tomorrow, and the guarantees stay the same. That is a much better property than "we use a good model", and it is only available to you if you own both ends.

Volume makes per-call pricing silly. A freelancer with five clients writes five of these a week, forever. The marginal cost here is electricity on a laptop that is already on.

And the reason I could do any of this: running locally meant I could read the raw output. Every finding in this post — the backwards version claim, the docs commit claimed as implementation, the three false flags in my own gate — came from looking at what the model actually returned, one line at a time, on a machine I control. An aggregate score would have hidden all of it.

Prize Categories

Best Use of Gemma. The writing layer is Google's gemma3:4b, served locally through Ollama on the CPU of an eight-core laptop with no GPU, and it is the reason this project exists in this shape. The model is good at the thing I am bad at — turning a scattered week into sentences — and unreliable in one specific, measurable way: it reaches for the most flattering verb that is still arguably true. Every rule in app/trace.py is a response to a failure I watched Gemma produce on real commits. Because it runs locally and costs nothing, I could reproduce those failures 36 times, read each one, and tell the difference between the thirteen the gate caught and the four it structurally cannot.

Every commit, sentence and evaluation in this post comes from synthetic fixture repos and my own projects. No client code was used, and no repository left the machine.

I poisoned 20 drug names to find out whether a local model belongs in a pharmacy

Sankalp Kotewar — Sat, 17 Oct 2026 04:22:57 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

A prescription reader for a neighbourhood medical shop in India. It runs on the shop's own laptop, with no network connection at all.

The shopkeeper photographs a handwritten prescription. The system reads it, works out what was actually prescribed, and decides for itself which lines are safe to fill and which ones need a human to look at the paper.

The person who would pay for this is the chemist himself. He currently re-reads every prescription by hand because he cannot afford to get one wrong, and he is doing it at 9pm after a fourteen-hour day.

I should be upfront that the interesting part of this project is not the reading. It is the deciding when not to trust the reading.

Demo

The whole pipeline runs from one command on the shop's machine:

python3 scripts/run_page.py prescription.jpg

A synthetic page, end to end:

--- OCR (1.77s wall, 1.64s engine, 11 observations -> 9 lines) ---
  [conf 1.00] Dr. R.Deshmukh MBBSReg2019/4482
  [conf 1.00] Pt: Mr. Suresh Wankhede 42/M
  [conf 1.00] Ox: HTN, T2OM
  [conf 1.00] Rx
  [conf 1.00] 1. Tab Telma 40mg 1-0-0 x 30 days
  [conf 1.00] 2. Tab Glycomet GP2 1-0-1 x 30 days
  [conf 1.00] 3. Cap Pan 40 1-0-0 x15 days
  [conf 1.00] 4. Inj. Insulin 0-0-1 x 10 days
  [conf 1.00] Adv: review after 15 days

--- reconciled prescription (57.1s wall) ---
  diagnosis: HTN, T2DM
  [0] Telma 40mg Tab sig=1-0-0 days=30 conf=0.70
  [1] Glycomet GP2 Tab sig=1-0-1 days=30 conf=0.70
  [2] Pan 40 Cap sig=1-0-0 days=15 conf=0.60
  [3] Insulin Inj sig=0-0-1 days=10 conf=0.60

--- gate: review ---
  - line 1 (Telma): model confidence 0.70 is below 0.80
  - line 2 (Glycomet): model confidence 0.70 is below 0.80
  - line 3 (Pan): model confidence 0.60 is below 0.80
  - line 4 (Insulin): model confidence 0.60 is below 0.80

total: 58.9s (OCR 1.8s + model 57.1s)

Look at the third OCR line and then the diagnosis line underneath it. Those are the same characters read two different ways: the OCR returned Ox: HTN, T2OM, and the model worked out that it was Dx: HTN, T2DM. That is the one thing here the model does genuinely better than any regex I could write.

The other thing worth noticing is that on this page, every line goes to a human. That is entirely the confidence threshold, not the interesting checks, because the model was candid enough to score its own repairs at 0.60–0.70. The structural checks are what earn their place on the poisoned set further down, where the model is not candid and hands back 0.95 on answers that are wrong.

On the timing: 57s is the steady state. The first page after the model has been idle takes about 80s, because gemma3:4b has to be loaded into memory first. I have not tried to hide that number. On an eight-core laptop with no GPU it is the real price of not sending the page anywhere.

A hosted copy, and a precise list of what it does not do

parchi-hub8.onrender.com

I want to be exact about the limits of that link, because the limits happen to be the argument.

It replays the real recorded OCR transcript from the page above — not a mock-up of one, the actual Vision output, including the Ox: HTN, T2OM line. The gate, the reorder forecast and the Hindi counselling text all execute live against it. If you want to see the gate refuse something, it is the line 3 (Pan) entry in the "Why" column.

What it does not do is run gemma3:4b. Render's free tier is a Linux container with 512MB of RAM and no GPU, and the model is 3.4GB. So the reading step is a deterministic fixture, and the page says so on itself rather than pretending.

Two consequences, stated rather than left for you to find:

The forecast on that instance reports "method": "damped-mean", not TabPFN. Torch does not fit there either; requirements-ml.txt is what switches TabPFN on, and the fallback announces itself in that field — which is the whole reason the field exists.
There is no audio on that instance, deliberately. The speech works: demo/counselling-hindi.mp3 in the repo is a real render, 39 seconds of Hindi, checked in so you can listen without a key. But the key is not set on a public host, because an open endpoint that spends credits on every request is a bill that anyone who finds the URL can run up. The key stays on the shop's machine.

That last point is the project's own thesis turned back on itself, so it seemed like the wrong thing to make an exception for.

Code

github.com/kotewar/parchi

The pieces worth reading:

app/gate.py — the deterministic safety checks, including the two that mattered most
app/gemma.py — the local model call, with the two API gotchas commented so nobody re-breaks them
app/tabpfn_engine.py — refill forecasting from the shop's own dispense history
scripts/selfcheck.py — 53 regression checks, runnable with plain python3, no model, no setup
scripts/eval.py — the poisoned-name eval harness
data/eval_poisoned.json — the 20 test lines and their ground truth
eval_base_v2.json and eval_primed_v2.json — the two runs quoted below, committed so the claims are checkable

How I Built It

I started by trying to get a vision model to read the handwriting. That was the wrong instinct, and the community had already measured why: a dedicated OCR engine is roughly 300x faster than a general-purpose vision model at reading, with fewer character errors (The Dedicated OCR Engine Lost to the General Purpose Model by hexisteme).

So the architecture splits the job:

page image
   -> macOS Vision OCR        (reading: 1.8s/page)
   -> gemma3:4b via Ollama    (meaning: what is this medicine?)
   -> deterministic gate      (safety: may this be filled?)
   -> human review            (only what the gate refuses)

The OCR reads. The model understands. The gate decides. Only one of those three is allowed to be wrong quietly, and it is not the model.

The measurement that started everything

While I was still validating the OCR path, I rendered a test prescription in a handwriting font and read it back. Vision read it flawlessly and reported confidence 1.00 on every line.

One of the lines it read perfectly was this:

Ox: HTN, T2OM

That is Dx: HTN, T2DM with two letters misread. Nothing about that line was illegible. It was simply wrong, and the OCR was certain.

That taught me the number I had been planning to trust measures "were the pixels legible", not "is this thing real". So I threw it out as a safety signal and went looking for what could replace it.

Confidence scores being useless for this is not a new observation, and I did not discover it. sohakanu documented it precisely while building Scribe: median confidence 0.95 when right, 0.95 when wrong (Confidence Is Theater). Their fix was cross-model verification — reading each field twice with models that fail differently. raju_dandigam makes the same point from the other direction in A Confidence Score Is Not a Probability.

I took a different route, because a pharmacy has something a form does not: a fixed vocabulary. The shop knows which forty medicines it actually stocks. That is checkable in code, for free.

So I wrote 20 poisoned drug names

I built a test set of twenty lines, each a realistic OCR corruption of a real Indian brand: 0 misread as O, 1 as l, 5 as S.

Then I ran them through gemma3:4b and sorted every answer into one of four buckets:

bucket	meaning
repaired	returned the true medicine
refused	declined to name one — safe
echoed_garbage	accepted the corrupted string as a medicine — dangerous
silent_wrong	invented a different medicine — dangerous

The last two are tracked together, because from the patient's side they are the same failure: a wrong medicine dispensed with confidence.

What the base model did

Nine of twenty lines came back wrong. All nine came back at 0.95 confidence.

Eighteen of twenty answers scored 0.90 or above. Nine of those eighteen were wrong. The confidence column carried no signal whatsoever, which means a threshold gate built on it would have passed nine wrong prescriptions straight through to the counter.

Two failure modes did the damage, and I only understood the second one by reading the eval output line by line.

It substituted generics for brands. Asked about Glycomet 6P2, it answered Metformin. For Pan 4O it answered Panadol. For Atorva 1O, Atorvastatin.

These are not cosmetic swaps. Glycomet GP2 contains glimepiride as well as metformin. Panadol is paracetamol; Pan is pantoprazole. A generic substitution is a different prescription.

And it is invisible in the JSON — except in one place. The name it returned is not anywhere on the page.

So I check exactly that. The longest word in the returned drug name gets compared against the raw OCR text with an OCR-sized edit tolerance:

def _grounding_problems(line: RxLine) -> list[str]:
    """Does the medicine the model named actually appear on the page?"""
    if not line.drug or not line.raw:
        return []

    drug_tokens = {t for t in tokens(normalise(line.drug)) if len(t) >= 3}
    if not drug_tokens:
        return []

    source_tokens = tokens(normalise(line.raw))
    anchor = max(drug_tokens, key=len)

    if any(closely_matches(anchor, source) for source in source_tokens):
        return []

    return [
        f"'{line.drug}' does not appear on the page as written "
        f"(OCR line reads: {line.raw.strip()!r})"
    ]

Checking that a value actually appears in its source is a known and documented technique — michael_maurice lays out the full discipline, including the crucial repairable/non-repairable split, in Schema-Valid Isn't Correct. I am applying it to brand names rather than invoice totals.

That one check caught all five substitutions the model made.

The second failure mode took me a whole eval run to see

With grounding in place, four of the nine were still getting through. All four were the same shape:

thyronorm-o-for-zero   strength='5Omcg'
ecosprin-s-for-five    strength='7S'
amlopres-s-for-five    strength='2.S'
insulin-units-word     strength='1O units'

Every one was the same thing. The drug name was correct, and it was genuinely on the page, so grounding had nothing to say about it. The strength was the corruption: 5O should be 50, 7S should be 75, 2.S should be 2.5, 1O units should be 10 units.

And my gate had accepted all four, because I was checking that the strength field was non-empty. Not that it was a strength.

Two regexes closed the gap. One for what a dose looks like, and one for the Indian combination markers that are legitimate strengths without being numbers at all:

_DOSE_RE   = re.compile(r"^\d+(?:\.\d+)?(?:mg|mcg|mgs|g|gm|ml|iu|units|unit|%)?$")
_SUFFIX_RE = re.compile(r"^[a-z]{1,3}\d+$")   # GP2 in Glycomet GP2

def _strength_problems(strength: str | None) -> list[str]:
    if not strength:
        return []

    compact = re.sub(r"\s+", "", strength)
    if _DOSE_RE.match(compact.lower()) or _SUFFIX_RE.match(compact.lower()):
        return []

    # If the digits are recoverable, say what we think it was. Suggesting a
    # correction is safe; applying one silently is not.
    guess = _repair_digits(compact)
    if guess != compact.lower() and _DOSE_RE.match(guess):
        return [
            f"strength {strength!r} does not parse as a strength "
            f"(looks like {guess!r} with a letter misread for a digit)"
        ]

    return [f"strength {strength!r} does not parse as a strength"]

Four misses went to zero. A wrong dose and a wrong drug are the same amount of wrong.

The false alarm I shipped, and how it surfaced

A word on false alarms, because a gate that cries wolf is a gate people click through. Three lines that the eval scores as safe still get sent to review: one where the model refused outright, one where it put x 30 days in the sig field, and the deliberately-illegible control case. I read all three. None is a correct answer being wrongly held back — they are the gate doing its job on a field my eval was not scoring.

Then, while reading the demo output above, I found one that was.

The strength check was flagging the GP2 in Glycomet GP2 as unparseable. GP2 is a combination marker — metformin plus glimepiride — and my regex only accepted digits with an optional unit. It was rejecting a correct answer, and no test caught it, because every test I had written encoded the same wrong assumption the code did. It surfaced because a human read the output.

That is now scripts/selfcheck.py: 53 checks covering the gate, the formulary, the counselling text and the end-to-end path, runnable with plain python3, no model and nothing to install. Two of those checks exist purely because of how the fix went wrong the first time — 1O units repaired to 10unit5 (the s in "units" was being read as a 5), and xyz acquired a confident repair hint for no reason at all. A wrong suggestion on a dosage is worse than no suggestion.

The same wrong assumption turned out to be sitting in the hosted demo's fixture too, a day later: the fixture had Glycomet GP2 with a null strength, so the gate reported "tablet with no strength" on a line whose strength was right there in the name. The real extractor returns GP2 in the strength field. The fixture was less realistic than the model it was standing in for, and it took reading the deployed page to notice.

Where it landed

	base model	with the shop's own correction history
wrong answers	9/20 (45%)	6/20 (30%)
correct	9/20	13/20
filled with no human looking	7/20	11/20
wrong answers reaching the counter	0/20	0/20

The model is still wrong a third of the time. None of it reaches the patient, because the gate does not accept the model's word for anything it can check itself.

The correction loop, and a result I did not expect

Every line the pharmacist corrects is written to a local SQLite table on the shop's machine. Those corrections are the shop's own vocabulary: in this shop, this handwriting, "Telrna" is Telma.

I fed 18 of them back into the prompt as context and re-ran the eval. It improved things: 45% wrong down to 30%, and four more lines filled without a human.

But it also introduced three regressions I would have missed if I had only looked at the totals:

glycomet-g-to-6     silent_wrong -> echoed_garbage   ("Glycomet 6P2" -> "Glycomet", GP2 dropped)
telmikind-h         repaired     -> echoed_garbage   ("Telmikind-H"  -> "Telmikind", -H dropped)
montck-dropped-e    silent_wrong -> echoed_garbage   ("Montck LC"    -> "Montck LC")

Given the shop's vocabulary, the model starts snapping to the nearest familiar name and silently drops suffixes that change what the medicine is. Telmikind-H contains a diuretic. Telmikind does not.

So the correction history is not a free win. It is a trade: better recovery of corrupted strengths, at the cost of a new way to be confidently wrong. The gate catches all three anyway, which is the point of having a gate that does not care how confident the model sounds.

This is also the honest reason the corrections matter. A hosted API cannot learn that this shop's handwriting turns Telma into Telrna, because the corrections are the most private thing in the building. They stay on the machine, and they are what a fine-tune would later be built from — via Tinker, so the shop never needs a GPU.

Why Does Open Innovation Matter?

Three reasons, in the order they actually matter to the shopkeeper.

The data cannot leave. A prescription carries a patient's name, age, diagnosis and medicines. That is the most sensitive record the shop holds. The whole point is that the model runs where the data already is. Shipping it to a hosted database to prove the model is local would give away the thing being protected.

The vocabulary is local knowledge. Knowing that this shop stocks forty specific brands, and that in this handwriting Telrna means Telma, is not general knowledge. It is not in anyone's training data. It exists only in that shop, and it is the difference between a system that works and a system that is confidently wrong. Owning the model is what makes the correction loop worth building.

Volume makes per-call pricing cruel. This is not a chat app. It is a shop doing a couple of hundred pages a day, forever. A per-page API bill is a recurring cost that grows with the shop's success. A local model costs electricity and a laptop the shop already owns — and the slowest thing in the pipeline is the part I did not pay for.

There is a fourth reason I did not expect. Running locally meant I could measure the model's failures on the shop's real material, as many times as I wanted, for free. The 20-line eval that this whole post is about cost nothing to run eleven times. That is not really possible when every run is a billed API call.

And a fifth, which is the one this post is really about. Running locally meant I could read the raw output. The generic-for-brand substitution at 0.95 confidence is not visible in an aggregate score. Neither was the false alarm I had shipped in my own gate. Both were found by looking at what actually came back, one line at a time, on a machine I owned.

The hosted demo makes the same point from the other side. Every capability I had to switch off to make it fit on a free container — the model itself, the forecaster, the voice — is a capability that works when the software sits next to the data and the patient.

My Agent Session

The bug I am least proud of, and the one that taught me the most: the false positive above was invisible to every test I had, because I had written the tests from the same wrong assumption as the code. Here is the session where reading the real output caught it.

Finding a false positive in a prescription safety gate

You

Continue if you have next steps, or stop and ask for clarification if you are unsure how to proceed.

Agent

I'd just refactored the formulary to share a tolerant matcher between the food-timing table and the chronic list. My test only re-ran the cases I had changed, which passed. Let me re-run the whole thing instead, on every entry.

It immediately blew up. food_timing("Glycomet GP2") raised KeyError. The shared matcher normalises hyphens to spaces and returns that normalised key, but FOOD_TIMING was still keyed "glycomet-gp2" with a hyphen. So it matched, then failed to index the table. Every hyphenated entry was broken.

▸ bash .venv/bin/python -c "from app.formulary import food_timing; food_timing('Glyc...

Input

$ .venv/bin/python -c "from app.formulary import food_timing; food_timing('Glycomet GP2')"

Output

KeyError: 'glycomet gp2'

-> root cause: match_key returns a NORMALISED key and indexes the table with it, but the table keys were not normalised

Fix: make the table keys normalised, and add an import-time guard so this fails loudly at import instead of at 9pm at the counter.

▸ edit app/formulary.py - rewrite FOOD_TIMING with normalised keys ("glycomet gp2", ...

Input

app/formulary.py - rewrite FOOD_TIMING with normalised keys ("glycomet gp2", "taxim o", "dolo 650") and add a guard that raises if any key is not already normalised

Output

Edit applied successfully

▸ bash .venv/bin/python -c "from app.formulary import food_timing, is_chronic; ..." ...

Input

$ .venv/bin/python -c "from app.formulary import food_timing, is_chronic; ..."  # every entry, not just the changed ones

Output

tables: 55 food-timing entries, 31 chronic
Glycomet GP2       food=with food      chronic=True
Pan 40             food=before food    chronic=False
Panadol            food=None           chronic=False
Taxim-O 200        food=after food     chronic=False
Mixtard 30/70      food=as the doctor has set out, with the injection

all reachable, and Panadol correctly gets no rule

Lesson worth writing down: re-running only the cases you touched is how a refactor ships a KeyError. The full sweep found it in one run.

Agent

Next I ran the actual demo page through the whole pipeline to get real output for the writeup, and read it line by line. That is where the more interesting bug was.

▸ bash .venv/bin/python scripts/run_page.py

Input

$ .venv/bin/python scripts/run_page.py

Output

--- OCR (1.78s wall, 1.64s engine, 11 observations -> 9 lines) ---
  [conf 1.00] Ox: HTN, T2OM
  [conf 1.00] 2. Tab Glycomet GP2 1-0-1 x 30 days

--- reconciled prescription (57.1s wall) ---
  diagnosis: HTN, T2DM
  [1] Glycomet GP2 Tab sig=1-0-1 days=30 conf=0.70

--- gate: review ---
  - line 2 (Glycomet): model confidence 0.70 is below 0.80; strength 'GP2' does not parse as a strength

total: 58.9s (OCR 1.8s + model 57.1s)

The gate is flagging GP2 as an unparseable strength. But GP2 is not a number at all -- Glycomet GP2 is metformin plus glimepiride, and the GP2 is the entire point of the prescription. The extraction prompt even lists "GP2" as a valid example of a strength. My regex only accepted digits plus an optional unit, so it rejected a correct answer.

This is the exact failure mode gate.py's own docstring warns about: a gate that cries wolf is a gate people click past, and then it protects nobody. And nothing in the repo caught it. It surfaced because a human read the output.

▸ read app/gate.py

Input

app/gate.py

Output

_STRENGTH_RE = re.compile(r"^\d+(\.\d+)?(mg|mcg|mgs|g|gm|ml|iu|units|unit|%)?$")
# A strength is digits, optionally a decimal, optionally a unit. Nothing else.

Fix: two branches. A dose -- digits, optional decimal, optional unit -- or an Indian combination suffix, one to three letters then digits. It costs nothing in detection, because every real corruption this check catches puts a letter where a digit should be ("7S", "5Omcg"), and the suffix branch accepts none of those.

▸ bash .venv/bin/python -c "from app.gate import _strength_problems; ..."

Input

$ .venv/bin/python -c "from app.gate import _strength_problems; ..."

Output

ACCEPTED: 40mg 500mg 2.5mg 10 units 75mcg GP2 gp2
FLAGGED:  7S -> looks like '75'
          5Omcg -> looks like '50mcg'
          2.S -> looks like '2.5'
          1O units -> does not parse
          6P2 -> does not parse

But two message-quality bugs fell out of the same reading:

1O units lost its repair hint. The letter-to-digit table maps "s" to 5, and "units" ends in s, so the repair produced "10unit5". Nonsense, and it threw away a hint that was correct.
xyz gained a hint. The map turned it into "xy2", which satisfies the new suffix shape and means nothing. A wrong suggestion on a dosage is worse than no suggestion.

▸ edit app/gate.py - split a trailing unit off before repairing digits; offer the re...

Input

app/gate.py - split a trailing unit off before repairing digits; offer the repair hint only when it lands on an actual dose

Output

Edit applied successfully

▸ bash .venv/bin/python -c "..." ; .venv/bin/python scripts/gate_recheck.py eval_bas...

Input

$ .venv/bin/python -c "..." ; .venv/bin/python scripts/gate_recheck.py eval_base_v2.json

Output

1O units -> looks like '10units'
xyz -> (no hint)

model dangerous  9/20
gate caught      9/9
gate MISSED      0/9
correct answers now over-flagged  0
END-TO-END dangerous rate 0/20 (0%)

So the fix removed a false positive without weakening detection: still 9/9 caught, 0 missed, 0 correct answers held back.

▸ write scripts/selfcheck.py - 47 checks, plain python3, no pytest, no model, no setup

Input

scripts/selfcheck.py - 47 checks, plain python3, no pytest, no model, no setup

Output

47/47 checks passed
gate, formulary and end-to-end checks all pass

selfcheck.py encodes every case that was wrong at some point, including this false positive, plus the formulary KeyError. It needs nothing installed, because a claim a judge cannot cheaply verify is not much of a claim.

Prize Categories

Best Use of Gemma. The understanding layer is Google's gemma3:4b, served locally through Ollama, running on the CPU of an eight-core laptop with no GPU. It is the only reason the "generic becomes brand" substitution is visible at all: the model reads Glycomet 6P2 and reaches for the only metformin brand it has ever heard of. Every safety check in the system exists because Gemma is useful and unreliable in a specific, measurable way — and because it runs locally, I could measure exactly how unreliable, eleven times over, for nothing.

Best Use of TabPFN. The shop's second problem is knowing what to reorder, and it is a tabular forecasting problem of exactly the kind a chemist cannot answer from memory: a dozen medicines, three months of dispense counts, and spiky demand around festivals. app/tabpfn_engine.py builds a weekly series per medicine and fits Prior Labs' TabPFN walk-forward — never training on a week it is about to predict — then ranks by urgency, weighted for the chronic medicines that must never run out.

It forecasts demand per medicine, not refills per patient. That is deliberate: the dispenses table has no patient column, because it is the only table that could leave the machine without carrying anyone's medical history with it. The richer signal would have cost the privacy claim the whole project rests on.

One detail I would keep in any system like this: without requirements-ml.txt the forecaster silently falls back to a damped moving average, so every row carries the method that produced it — "method": "tabpfn (53 shop rows)". A fallback can never be mistaken for a forecast. That field is also how I discovered that tabpfn was not installed at all while the code claimed it was, which is a lesson about the difference between integrating a library and running one.

Best Use of ElevenLabs. The last mile of a prescription is the part the pharmacist has already said fifty times before lunch: what to take, when to take it, for how long, and what to do if something goes wrong. That explanation is spoken, not read. Many of the people this shop serves would rather hear it than read a label, and by the fiftieth time the pharmacist is no longer saying all of it.

ElevenLabs' eleven_multilingual_v2 renders that script in Hindi from prompts/counsel.txt, and the parts he drops are exactly the parts written into the prompt: never take two doses at once to catch up on a missed one, come back to the shop with the strip if anything is unclear, and see a doctor immediately for a rash, swelling or difficulty breathing. It is a real render — 485 characters in, 39 seconds of Hindi out, at one credit per character — and it is checked into the repo as demo/counselling-hindi.mp3 so it can be listened to rather than taken on trust.

One thing I got wrong first and would have shipped: I originally rendered it with a default English premade voice. eleven_multilingual_v2 will happily make an American voice read Devanagari script, and it sounds exactly as wrong as that suggests — a patient being told their dosage in a foreign accent. Auditioning the Hindi voices in the shared library and switching to a native one cost 240 credits and one afternoon. Worth saying out loud, because "the multilingual model speaks Hindi" and "the voice sounds like it speaks Hindi" are two different claims, and only the first one is the model's job.

The design decision I would defend is that the audio is opt-in per request and off by default. It is the only step in the pipeline that sends anything off the machine, and what it sends necessarily contains the medicine names — a patient cannot be told to take something the sentence refuses to name. So the text stays local and the pharmacist asks for the speech. The hosted demo has no key at all, for the same reason.

Best Use of Render. The hosted demo above is deployed from a Render blueprint — render.yaml lives in the repository, so the service is described in version control rather than clicked together in a dashboard, and a stranger can reproduce the deploy from the repo alone.

The interesting part was making a free-tier container honest. There is no GPU and 512MB of RAM, so instead of pretending, the deployment was designed around the constraint: requirements.txt and requirements-ml.txt are separate files so the deployable dependency set stays small enough to build without dragging in torch, app/main.py never imports the macOS Vision wrapper (app/ocr.py) so the Linux container boots, and DEMO_MODE=true replays the real recorded OCR transcript while the gate, the forecaster and the counselling text all execute for real. The page tells you which parts are fixtures and why, which is the only reason I would put a demo link in a post about not trusting things.

Thanks to sohakanu, michael_maurice, raju_dandigam and hexisteme — this project would have taken a much longer route to a worse place without their write-ups.

All names, prescriptions and corrections in this post are synthetic. No real patient data was used, and no page left the machine.

ConcreteOasis: Screenless Urban Microclimate & Heat-Island Guide Powered by Gemma 4 & TabPFN

Sankalp Kotewar — Tue, 06 Oct 2026 18:52:07 +0000

A zero-touch, screenless ambient walking guide that turns urban concrete jungles into mindful green escapes using Gemma 4, TabPFN, and ElevenLabs.

The Paradox: Touching Grass in a Concrete Jungle

Over four billion people live in dense urban centers. When hackathons announce themes like "Touch Grass" or outdoor navigation, the default assumption is that you live a 15-minute drive away from an alpine hiking trail or national park.

For most of us working tech jobs in bustling cities, that is not reality. Our outdoors is the concrete jungle: sun-baked asphalt roads, narrow colony lanes, flyovers, and scattered municipal green pockets.

Worse yet, the moment we step outside to decompress, existing fitness and trail apps do the exact opposite of helping us disconnect: they vibrate constantly, demand screen taps, display noisy dashboards, and force us to look down at glass instead of up at our surroundings.

We built ConcreteOasis to solve this. It is an ambient, zero-touch walking companion that runs silently from your pocket. By combining Gemma 4, Prior Labs' TabPFN, and ElevenLabs, ConcreteOasis predicts microclimatic heat islands and pavement hazards on everyday city streets—whispering terse, mindful cues into your earphones only when conditions shift.

Total screen time required during a 35-minute urban walk: 0 seconds.

What We Built

ConcreteOasis turns any smartphone and basic earphones into an autonomous, screenless urban nature guide:

Zero-Touch Pocket Mode: You tap "Start Walk" at your front door, and the web client engages the Screen Wake Lock API over a pitch-black OLED canvas (#000000). Your phone stays safely stowed in your pocket with zero battery burn and zero risk of mobile operating systems suspending background GPS.
Microclimate & Urban Heat-Island Prediction: Standard weather forecasts give a single, blanket city temperature. But asphalt radiates up to 8°C higher than ambient air, while a dense Neem or Peepal canopy provides an immediate micro-cooling pocket. Prior Labs' TabPFN computes hyper-local surface heat deltas and pavement slickness probabilities in real time from tabular environmental features.
Terse Conversational Filtering via Gemma 4: Instead of robotic GPS chimes or raw sensor telemetry, Gemma 4 acts as an intelligent conversational filter. If conditions are unchanged, it enforces silence. When you enter a shaded pocket or hazardous stretch, it synthesizes a grounding audio cue in under 15 words.
Hands-Free Ambient Audio via ElevenLabs: The synthesized cue streams directly to your Bluetooth earbuds as a calming natural voice.

Why Open-Source AI Matters

Every prompt question for the Hacktoberfest Open-Source AI Challenge guided our architectural decisions:

Data Sovereignty & Privacy: Personal location telemetry, stride cadence, and movement logs are deeply sensitive. With open-weight models like Gemma 4 and self-hosted agents, your daily routes and health habits are never fed into proprietary cloud advertising datasets.
Specialized Tabular Foundation Models Over Monolithic LLMs: LLMs notoriously hallucinate when asked to perform arithmetic on continuous multi-variable physics data. By using Prior Labs' TabPFN, we perform zero-shot Bayesian tabular inference on microclimate metrics (soil moisture, canyon aspect ratio, solar elevation) with mathematical precision.
Freedom from API Tolls: Gemma 4's Apache 2.0 licensing and ultra-efficient edge footprint (E2B / E4B with Per-Layer Embeddings) prove that personal, ambient AI can run cost-free on everyday developer hardware without recurring API subscription barriers.

Architecture & How It Works

[IN POCKET: PHONE / WATCH]
Smartphone (OLED Screen Wake Lock, Background GPS Polling)
  │
  │  HTTP POST /api/telemetry { lat, lon, speed, heart_rate }
  ▼
[RENDER CLOUD HOST: FastAPI Gateway]
  │
  ├── 1. Context Aggregator:
  │      Pulls Open-Meteo microclimate metrics & OpenStreetMap street canyon attributes.
  │
  ├── 2. TabPFN Inference (Prior Labs):
  │      Predicts [heat_island_delta_celsius, pavement_slip_risk] from tabular vector.
  │
  ├── 3. Gemma 4 Agent (Ollama / Local Endpoint):
  │      Synthesizes situational context into a <=15 word spoken whisper (or returns SILENCE).
  │
  ├── 4. ElevenLabs Voice API:
  │      Converts text cue into streaming audio buffer sent to earphones.
  │
  └── 5. Sentry Agent Tracing:
         Distributed spans track telemetry ingestion, TabPFN latency, and token generation.
  │
  ▼
[EARPHONES / AIRPODS]
Ambient audio cue plays automatically over Bluetooth.

Technical Highlights & Implementation

1. TabPFN Microclimate Prediction (`tabpfn_engine.py`)

import numpy as np
import pandas as pd
from tabpfn import TabPFNRegressor, TabPFNClassifier

class UrbanMicroclimateEngine:
    def __init__(self, training_csv: str):
        df = pd.read_csv(training_csv)
        X = df[["surface_type", "solar_angle_deg", "canopy_pct", "rain_48h_mm", "canyon_ratio"]]

        self.heat_regressor = TabPFNRegressor()
        self.heat_regressor.fit(X, df["heat_island_delta_celsius"])

        self.slip_classifier = TabPFNClassifier()
        self.slip_classifier.fit(X, df["pavement_slip_hazard"])

    def evaluate_location(self, features: list) -> tuple[float, float]:
        X_eval = np.array([features])
        heat_delta = float(self.heat_regressor.predict(X_eval)[0])
        slip_prob = float(self.slip_classifier.predict_proba(X_eval)[0][1])
        return round(heat_delta, 1), round(slip_prob, 2)

2. Gemma 4 Conversational Synthesis Prompt

System: You are ConcreteOasis, an ambient walking companion for city streets.
Synthesize tabular microclimate metrics and urban trees into an ultra-concise spoken whisper.
Rules:
- Maximum 15 words.
- Calm, observant, natural tone.
- If neither thermal relief nor significant greenery nor slip risk is present, output exactly: SILENCE.
- Never output markdown, explanations, or filler.

Input: {"canopy": 0.85, "tree": "Neem", "heat_delta": -3.4, "slip_risk": 0.12, "cadence": "steady"}
Output: Stepping under dense Neem canopy. Ambient heat drops three degrees. Ease your stride and breathe.

3. Distributed Observability with Sentry Agent Tracing

import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration

sentry_sdk.init(
    dsn=settings.SENTRY_DSN,
    traces_sample_rate=1.0,
    integrations=[FastApiIntegration()]
)

@app.post("/api/telemetry")
async def handle_telemetry(payload: TelemetryPayload):
    with sentry_sdk.start_transaction(op="agent.cycle", name="UrbanStrideCycle"):
        with sentry_sdk.start_span(op="tabpfn.inference", description="Predict microclimate"):
            heat_delta, slip_risk = tabpfn_engine.evaluate_location(features)

        with sentry_sdk.start_span(op="gemma.synthesis", description="Gemma 4 text synthesis"):
            cue_text = await gemma_agent.generate_ambient_cue(...)

        if cue_text != "SILENCE":
            with sentry_sdk.start_span(op="elevenlabs.tts", description="Stream voice"):
                audio_bytes = await audio_service.synthesize_speech(cue_text)

Field Test: Taking ConcreteOasis Outside

To prove that the screen was truly the shortest part of the experience, we took ConcreteOasis out for a 35-minute evening walk through an urban neighborhood:

Setup: AirPods in ears, tapped "Start Walk" on the phone, and stowed the phone into a pocket.
The Experience:
Crossing an open asphalt intersection: complete silence. The app didn't bombard us with meaningless step counters.
Turning into a shaded residential avenue lined with mature trees: The earphones chimed softly:

"Entering dense shade cover. Temperature drops three degrees. Keep this easy stride."
Approaching an alleyway with broken, wet paver stones: TabPFN flagged an 81% slip probability, prompting:

"Uneven, damp paving stones ahead from morning runoff. Watch your footing on the bend."
The Verdict: Zero glances at a screen for the entire 35 minutes. We returned home relaxed, having engaged with our surroundings rather than a glass display.

Built with GitHub Copilot in VS Code

We scaffolded and developed ConcreteOasis entirely in VS Code using GitHub Copilot:

Agent Scaffolding: Used Copilot Chat to design the Pydantic schemas and tie the asynchronous FastAPI pipeline together.
Geofencing & Tabular Matrices: Copilot generated the feature matrix normalizers and synthetic training distribution for TabPFN.
Client Audio Buffer: Copilot wrote the Web Audio context handler that allows uninterrupted background Bluetooth audio playback on mobile browsers.

Prize Categories Entered

Best Use of Gemma ($200): Gemma 4 serves as the core natural language reasoning engine, filtering alerts and generating concise spoken wisdom.
Best Use of TabPFN ($200): Prior Labs' TabPFN tabular foundation model predicts continuous urban heat-island deltas and pavement slip classification.
Best Use of Render ($200): The FastAPI edge backend and PWA client are deployed on Render.
Best Use of GitHub Copilot ($100): Scaffolded end-to-end inside VS Code using Copilot Chat and agent mode for geofencing math and tool schemas.
Best Use of ElevenLabs ($100): Natural streaming audio delivery directly into wireless earphones.
Best Use of Sentry Agent Tracing ($100): Instrumented end-to-end spans showing TabPFN inference time, Gemma token latency, and audio delivery roundtrips.

Demo & Repository

Live Demo: TBD
GitHub Repository: TBD

DEV Community: Sankalp Kotewar

The AI didn't invent my client update. It promoted it.

What I Built

Demo

Code

How I Built It

Start with the failure you can actually catch

The traps are the test

What it caught

What it measures

Three bugs the output found, not the tests

The Lazy Payoff

Why Does Open Innovation Matter?

Prize Categories

I poisoned 20 drug names to find out whether a local model belongs in a pharmacy

What I Built

Demo

A hosted copy, and a precise list of what it does not do

Code

How I Built It

The measurement that started everything

So I wrote 20 poisoned drug names

What the base model did

The second failure mode took me a whole eval run to see

The false alarm I shipped, and how it surfaced

Where it landed

The correction loop, and a result I did not expect

Why Does Open Innovation Matter?

My Agent Session

Prize Categories

ConcreteOasis: Screenless Urban Microclimate & Heat-Island Guide Powered by Gemma 4 & TabPFN

The Paradox: Touching Grass in a Concrete Jungle

What We Built

Why Open-Source AI Matters

Architecture & How It Works

Technical Highlights & Implementation

1. TabPFN Microclimate Prediction (tabpfn_engine.py)

2. Gemma 4 Conversational Synthesis Prompt

3. Distributed Observability with Sentry Agent Tracing

Field Test: Taking ConcreteOasis Outside

Built with GitHub Copilot in VS Code

Prize Categories Entered

Demo & Repository

1. TabPFN Microclimate Prediction (`tabpfn_engine.py`)