DEV Community: Gajanand V Dhayagode

My fine-tuned Gemma blamed a critical haemoglobin on "how the sample was handled". My eval gave it a pass.

Gajanand V Dhayagode — Mon, 19 Oct 2026 06:58:31 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 2 — Side Hustle.

What I Built

In India, your lab report usually arrives as a PDF on WhatsApp, hours or days before you see a doctor. So people ask the health professional who's always there, the pharmacist at the counter: "Is this bad?"

Report Saathi (saathi means "companion") answers that question. You upload the PDF (or a photo), say whether you're pregnant, and within a few minutes (about one for a short report, three to five for a full check-up) you get a short explanation in plain English:

the values worth asking your doctor about, and what each test measures;
small differences grouped in one line, so a full check-up shows a handful of things instead of sixteen alarms;
the lab's own notes, quoted as the lab's words;
neutral questions to take to your doctor;
a "contact your doctor today" line, only for critical values.

It never diagnoses and never mentions medicines. Every answer ends: "This explains your report. It is not a diagnosis. Please show this report to your doctor."

The side hustle: a pharmacy puts a QR code on the counter. Customers scan it and upload their report, and each upload is counted for that shop. The shop pays a small monthly fee. The customer gets an answer, and the pharmacist gets their time back.

The niche: pregnancy. Pregnancy changes what "normal" means. Haemoglobin, TSH and the sugar test all have different ranges in each trimester. An explainer that uses adult ranges will tell a pregnant woman her normal result is low. Report Saathi asks about pregnancy (reports never say), uses the trimester range, and tells her when a value is normal only because she's pregnant.

This post is mostly about one lesson, though: a reassuring excuse is worse than no reason at all, and my own fine-tune taught me that.

Demo

See a finished example right away (no upload, no waiting): https://report-saathi.onrender.com/example
It walks through the three steps on a made-up report: what the reader found, what the rules decided, and the explanation.

Try your own report: https://report-saathi.onrender.com

Measured on the live app with real (redacted) full check-ups of 83–87 rows: read in 117–150 s, explained in 69–126 s, 3–5 minutes in all. A short report takes about a minute.

Code

gajanand27-05 / report-saathi

Report Saathi: understand your lab report

Report Saathi (saathi means "companion") explains a blood-test report in simple English. You upload the PDF your lab sent on WhatsApp, or a photo of the paper report, and say whether you're pregnant. A few seconds later you see:

which values are worth asking your doctor about, and what each test measures
small differences that usually don't matter on their own, grouped in one line
your normal results in one line
the lab's own comments, quoted as the lab's words
neutral questions to ask your doctor
an urgent "contact your doctor today" line, only for critical values

It never diagnoses and never suggests medicines, supplements or doses. Every answer ends with: "This explains your report. It is not a diagnosis. Please show this report to your doctor."

Built for the DEV "Hacktoberfest Open-Source AI Challenge: Side Hustle" (15–19 October 2026). Measured…

View on GitHub

The web app deploys to Render with one click (free plan, no card). The worker runs on any machine with an OpenAI-compatible server (we use vLLM on a DGX Spark). rules.csv, the table that decides what's high or low, is plain text that a doctor can check line by line.

How I Built It

phone ──upload──► web app on Render ◄── polls (outbound only) ── worker on a DGX Spark
                  job queue (Postgres)                            1. READER     Gemma 4 12B reads the report
                  deletes everything after 24 h                   2. RULES      rules.csv flags each value (no AI)
   ◄── explanation streams back ───┘                              3. EXPLAINER  the same 12B + our LoRA explains

The first decision was the important one: no AI decides whether a value is abnormal. A model that says "your TSH is fine" when it isn't is the worst thing this app could do. So the job is split in three, and the AI never makes the medical call.

1. The reader has to prove every number

Indian lab PDFs usually have a text layer, but it's jumbled: columns interleave, ranges float above their test, and one test can print five ranges (child, adult, three trimesters). So Gemma 4 gets each page as an image plus that page's exact text, and returns the rows.

Then one rule makes it trustworthy: every number Gemma returns must appear in the PDF's own text. If it doesn't, the row becomes "couldn't read" and is never explained. Photos have no text layer, so the app shows the numbers it read and asks "Does this match your report?" before explaining anything.

On 27 real reports (names, IDs, doctors and lab removed; used only for testing, never for training), 969 rows in all, I tested two readers:

	Gemma 4 31B	Gemma 4 12B (live)
High/low results the user sees	960 / 960 right	958 / 960 right
Values exactly right	963 / 969	966 / 969
Values it made up	1, caught by the PDF check	0
Time per report	median 37 s	median 110 s

The 31B reads better and faster. The live app uses the 12B anyway, because one model does both jobs: the 12B's own weights read the report, and the same server applies our LoRA to write the explanation. The 31B was only ever the teacher, and the GPU box is shared with other projects, so one 12B instead of two big models frees about 46 GB for everyone else.

The cost is real and I'd rather show it: one of the 12B's two misses turned a low HDL into "Normal", most likely because it misread the printed range on a crowded page. I tried a safety net that cross-checks every value against our own table, but on correct reports the two disagree on 2% of rows (mostly small red-cell indices), so it would have raised false alarms. That's the first thing to fix next.

2. The rules: a table a doctor can check

rules.csv lists every test the way labs print it ("SGPT", "ALT (SGPT)"…), with units, ranges, pregnancy ranges by trimester, critical limits and an importance level. The rules use the lab's printed range first, pick the one that matches the patient's age or trimester, and handle bands ("5.7–6.4 = pre-diabetic") and values that fall in the gap between them.

The real reports kept teaching it. The first 6 were thyroid, HbA1c and antenatal panels. The next 21 brought urine tests, dengue, typhoid (Widal), HIV and hepatitis screens and urine cultures: 194 rows the table didn't know. Now it knows all of them, with care. A positive screening test is "a screening result your doctor will confirm", never "you have HIV". A culture report keeps the germ's name but drops the antibiotic list, so the explainer never sees a medicine name.

The rules also gave me the most embarrassing bug of the week. This lab prints platelets as "373 lakhs/cmm", which would be 100× too many. It means 373 thousand. My rules converted it correctly to 3.73 lakh, then attached a note saying "read as thousand/cmm". Put that next to 3.73 and a patient reads 3,730: a dangerously low count. The rules were right and the note was wrong. The blind review below caught it; the note now gives both numbers.

3. The explainer, and the excuse it learned

The explainer only sees the flagged rows. I wanted a small, fast model, so the plan was Gemma 4 31B as the teacher and Gemma 4 12B as the student, fine-tuned on the team's DGX Spark.

The base 12B wasn't safe enough. It skipped the doctor questions on all three critical cases in my first test set, and it told two patients they had a "urinary tract infection". That's a diagnosis, which this app must never make.

So I made training data. A generator writes synthetic reports in real lab formats (40% pregnant, lots of critical values and odd cases). The rules flag them and the 31B explains each one. An answer is kept only if it passes every automatic check (no diagnosis, no medicine words, every flagged number mentioned, the urgent line only when critical) and a review by the 31B as a judge. Real reports are never used.

Fine-tune 1 copied the teacher's filler. It learned the format perfectly, and it also learned "this can happen for several general health reasons." True, and useless. So I wrote a rule against vagueness: give "concrete, everyday, non-alarming reasons", for example a recent meal, dehydration, the time of day, how the sample was handled. I added the same examples to the judge.

Fine-tune 2 looked like a win. 2,351 of 3,500 teacher answers passed, ft2 trained on 2,000 of them (a 16-bit LoRA on gemma-4-12B-it), and on my 36-case test (each case run twice) it beat everything:

Model	Passes checks + judge (of 72)	Rule breaks
Gemma 4 12B, base	39	6
Gemma 4 31B, base (the teacher)	53	9
12B, fine-tune 2	63	0

A fine-tuned 12B beating its own teacher, with zero rule breaks. I nearly made that the headline.

Then the answers were read blind. I gave 13 shuffled pairs (fine-tune 2 vs the 31B, same report) to a separate AI model as a reviewer, without telling it which was which. The 31B won 9, the fine-tune 3, one tie. The reasons were the problem:

Haemoglobin 6.2, critically low, 32 weeks pregnant (fine-tune 2): "This can happen due to the body's increased needs during pregnancy or **how the sample was handled."
A positive HIV screening test (fine-tune 2): "This can happen **if the sample was handled in a certain way* or if there is a false positive result."*
An HbA1c just above normal, a 3-month average of blood sugar (fine-tune 2): "This can happen due to **a recent meal, the time of day* the sample was taken…"*

All three passed my eval. They're calm, specific and everyday, which is exactly what I'd asked for. The HbA1c one is simply false: a 3-month average doesn't care about breakfast. The other two are worse. They hand a frightened person the comfortable reading of a result that needs a doctor today.

My example list had become the answer. And the student learned it harder than the teacher. Counting the excuses in the same 72 answers:

	"time of day"	"how the sample was handled"	"all other tests were normal" on a one-test report
Gemma 4 12B, base	0	11	1
Gemma 4 31B (teacher)	16	18	18
12B, fine-tune 2	30	27	20

A fine-tune doesn't just copy the teacher. It amplifies whatever the filter rewards, and my filter rewarded reassurance. The judge was the 31B grading with my examples, so it marked the excuses as good answers. Scoring 63 of 72 measured how well ft2 had learned my rubric, not whether it explained well.

The fix was to the rubric, not the model. Prompt v1.4 says: give a real, common cause for that test, or none at all. For example, low MCV is commonly linked to low iron or an inherited trait; HbA1c isn't changed by a meal. Critical values and positive screening tests get no reassuring reason. The urgent line moves to straight after the overview. New automatic checks catch "how the sample was handled", meal excuses for HbA1c, and invented "all other tests are normal" lines. The judge now fails a wrong reason instead of rewarding it.

Then I re-tested both models with prompt v1.4 and the stricter judge (same 72 answers, no retraining):

	Passes (of 72)	Rule breaks	"time of day"	"sample"	invented "all other tests normal"
Gemma 4 31B	54	2	0	2	4
12B, fine-tune 2	64	0	0	2	0

The same fine-tune, given honest instructions, dropped the habit it had learned hardest: "time of day" went from 30 answers to 0. The critical haemoglobin now reads:

"Please contact your doctor today about your Haemoglobin." (straight after the overview), then: "Haemoglobin… is 6.2 g/dL. This is critically low. The normal range is different for people in their third trimester of pregnancy."

No excuse at all. And the low MCV that used to be blamed on the time of day now gets "Small red cells are commonly linked to low iron." The 31B's two rule breaks were both "urinary tract infection" on urine tests, a diagnosis. So fine-tune 2 writes the explanations in the live app, with prompt v1.4.

A second blind read of these new answers is next.

There wasn't time to retrain before the deadline. A fine-tune 3 on v1.4 data is next, and this time I'll read its answers blind before I believe its score.

Speed: Gemma 4's FP8 serving with its MTP drafter makes the 12B 5× faster (3.4 s against 17 s per explanation). It cost the base 12B nothing in quality, but it cost the fine-tune 4 passes and 3 rule breaks, so I didn't take the trade. The text streams in as it's written anyway.

One deploy surprise

I planned a paid Render setup with an hourly clean-up job. My Indian debit card was declined ("does not support this type of purchase"), so it runs on Render's free plan. With no cron, the app deletes jobs older than 24 hours itself whenever the worker checks in, and the worker's polling keeps the free service awake. The free Postgres worked on the first deploy.

What still doesn't work

The reasons look fixed on the test, but the habit is still in the weights. Prompt v1.4 holds it back; a fine-tune 3 trained on v1.4 answers would remove it. Two answers still mention the sample.
Counting on big reports. On a 70-value check-up the models sometimes miscount what's worth asking about.
No doctor has reviewed it yet. A gynaecologist is reviewing the pregnancy ranges and critical limits. Until then, rules.csv is carefully researched, not clinically signed off.
The 12B is slow on big reports. A full 84-row check-up takes about 2 minutes to read and 1–2 minutes to explain (3–5 minutes in all on the live app); the explanation streams in as it's written. Serving the 12B in FP8 with Gemma 4's drafter would be about 5× faster, which I measured, but the fine-tune lost quality in FP8.
A real report broke the live app twice last night (once on the explainer's context limit, once on a missing range label in the rules). Both are fixed on the GPU box; the second fix still has to reach the public code.
One lab. All 27 real reports use one Bangalore lab's template. Other labs will break the reader in new ways.

I built this with Claude Code as my pair programmer. A second Claude Code session ran the DGX through a private repo: I wrote task files, it pushed results. So every number above came from a script, and the one that fooled me came from my own rubric.

Why Does Open Innovation Matter?

Lab reports are some of the most private documents people have. They show pregnancies, HIV tests and diabetes. With open weights, every model runs on a machine our team owns. No report goes to an AI company, and Render deletes everything within 24 hours.
I could find the bug in my own rubric. With open weights I could train my own judge-filtered data, count every excuse in every answer, and compare the student with its teacher. A closed API would have handed me a score, not a habit to fix.
It costs nothing per report. Thousands of teacher answers, two fine-tunes and six-model evaluations, with no API bill. That's also what makes a small monthly fee per pharmacy workable.
The honest trade-off: a big closed model would probably have given better medical reasons on day one. I chose privacy and paid for it in engineering. This week, that engineering is the story.

Prize Categories

Best Use of Gemma. Gemma 4 does all the AI work. In the live app one Gemma 4 12B does both jobs: its own weights read every report (page image plus text, each number checked against the PDF), and our 16-bit LoRA on the same server writes every explanation. Gemma 4 31B taught and judged the fine-tune. We fine-tuned Gemma 4 12B with a 16-bit LoRA, measured FP8 and the MTP drafter, and found out, by reading blind, what the scores were hiding.
Best Use of Render. Render hosts the front end, the job queue (web service + Postgres from one render.yaml, free plan) and an example page that works even when the GPU box is busy. The GPU worker only ever connects outward to Render, so it needs no open ports.

Got a lab report on your phone right now? That's exactly who this is for. And if you're grading a model with its own teacher, read some of its answers blind. 🩺

My AI field guide called castor a "Virginia Creeper". A 107-line list fixed more than fine-tuning did.

Gajanand V Dhayagode — Sun, 11 Oct 2026 17:48:05 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 1 — Touch Grass.

What I Built

Pocket Naturalist is an offline field guide for walks. You photograph a plant, bird or bug, and Gemma 4 12B, running on my own laptop, tells you what it probably is. It also points out one detail to look at more closely, gives you a small quest to find next ("find a plant with compound leaves"), and writes a journal of your walk at the end.

There's no cloud and no account, and it works with no mobile signal: the phone talks to the laptop over a hotspot. You can snap photos live, or, if you don't want to carry a laptop around a garden (I didn't), take photos with your normal camera and add them from the gallery afterwards.

The rule I gave myself: the screen should be the shortest part of the walk. The answer is read aloud, and the phone goes back in your pocket.

What it does:

👀 Shows its reasoning: the features it saw, its best guess, and what else it could be.
🔁 Takes a second look when it's unsure: it describes the plant's structure first, then names it.
🔍 Mystery finds: when it's still unsure, it gives you one thing to check on the real plant ("blue flowers mean butterfly pea…") instead of a confident wrong answer.
✏️ "I know what this is": correct it, and the journal gives you the credit.
🌿 An Indian field guide: the fix that made it actually useful where I live. More on that below.

Demo

Two weeds Gemma got wrong before the field guide. Notice the "look closer" tips: they only describe what's actually in the photo.

It was "Sago Palm" before the field guide.

When Gemma isn't sure, it doesn't guess. It tells you what to check on the real plant.

The walk journal, written by Gemma from my 17 finds. It uses Gemma's own names, mistakes included (that's not teak), which is exactly why the app shows "could also be" and lets you correct it.

On my real 17-photo walk, here's what changed after the fix:

Plant	Stock Gemma 4	With the field guide
Castor	Virginia Creeper	✅ Castor
Curry leaf	Chinese Fringe Tree	✅ Curry leaf
Congress grass	Common Ragweed	✅ Parthenium
Four o'clock flower	"Mexicaner" (not a plant)	✅ Four o'clock flower
Black nightshade	Wild Jasmine	✅ Black nightshade
Araucaria	Sago Palm	✅ Norfolk Island / Cook pine
Red rose	Adenium	✅ Rose

Overall: 4 → 10 correct out of 15 photos where I know the right answer.

Code

gajanand27-05 / pocket-naturalist

Offline field guide for walks: snap a plant or bird, local Gemma 4 identifies it and sends you on the next quest. Hacktoberfest 2026 Touch Grass.

🌿 Pocket Naturalist

An offline field guide for walks. Snap a plant, bird or bug on your phone. Gemma 4, running on a laptop in your backpack, tells you what it probably is, what to look at more closely, and gives you a small quest to find next. When you finish, it writes a journal of your walk.

You don't need mobile signal, a cloud API or an account. Your photos never leave your own devices.

Built for the Hacktoberfest 2026 Open-Source AI Challenge, Week 1: "Touch Grass".

How it works

phone (browser) ──same Wi-Fi or hotspot, no internet──▶ laptop: server.py ──▶ Ollama ──▶ gemma4:12b (vision)

Start a walk. You get your first quest, read aloud by the phone's built-in offline voice.
📷 Snap a find, or 🖼️ add photos from your gallery later. You don't have to carry the laptop: take photos in the garden with…

View on GitHub

It's one standard-library Python file plus one HTML page, with nothing to pip install:

ollama pull gemma4:12b
python server.py   # prints a URL to open on your phone

How I Built It

The setup: phone browser → Wi-Fi hotspot (no internet) → server.py on my laptop → Ollama → gemma4:12b. The laptop is nothing special: an RTX 3050 with 6 GB of GPU memory and 16 GB of RAM. Gemma 12B doesn't fully fit, so it runs partly on the GPU and partly on the CPU, at about 30–40 seconds per photo. Every answer is forced into a fixed JSON format with Ollama's format option, so the app never has to guess what the model meant.

Day 1: making it honest

The first thing I learned is that a vision model can be confidently wrong. A house sparrow came back as "Jungle Babbler, 95% sure." A "fun fact" about hibiscus was made up, and so were some of the scientific names. So I:

made Gemma list what it sees first, then rank three candidates, then choose;
removed fun facts and scientific names and replaced them with "look closer" tips that may only describe what's visible in the photo;
told the journal to use only the walk data. Its first draft described "the afternoon sun hanging low" on a walk it knew nothing about.

Then a vine on my railing came back as "Creeping Charlie". I went outside to look: it had bright blue flowers. Butterfly pea. That moment became a feature: when Gemma isn't sure, the app now gives you a clue to check on the real plant instead of a guess. That sends you back to the plant instead of back to the screen, which is the whole point of "Touch Grass".

Day 2: the real walk, and a bias I didn't expect

I walked my neighbourhood and took 17 photos of ordinary plants: castor, moringa, curry leaf, a four o'clock bush, roadside weeds. Stock Gemma got 4 of 15 right.

The interesting part was how it was wrong. For the castor plant, Gemma described "palmately lobed leaves, deeply serrated edges, large star-shaped leaf". That's a perfect description of castor. Then it named it "Virginia Creeper", an American vine. It did the same everywhere: moringa became "Purslane", curry leaf became "Chinese Fringe Tree", and a four o'clock flower became "Mexicaner", which isn't a plant at all.

Its eyes were fine. Its vocabulary was American.

Fix 1: a field guide in plain text

Real field guides are regional, so I wrote one: field_guide.csv, a list of 107 common Indian plants from gardens, kitchens, roadsides, waste ground and parks. Each entry has the features that tell it apart. For example:

Castor: shrub; very large star-shaped palmate leaves with 7–11 toothed lobes; reddish stems; spiny seed clusters

It goes into Gemma's prompt with three rules I learned the hard way:

Sort it A–Z. In my first version, "Holy basil" was the first entry, and Gemma started calling half my photos holy basil.
Plants only. Without that rule, the house sparrow became a "lark".
"Never force a match." If nothing fits, give your own best name with lower confidence.

I also had to keep the prompt short enough to fit Ollama's default 4,096-token context. When I raised it, my 16 GB laptop ran out of memory and the app server was killed. Dropping the local-language names from the prompt (they stay in the CSV for people) fixed that.

Measured on 28 labelled photos (22 of mine, 6 from the internet):

	Correct (first answer)	My garden photos
Stock Gemma 4 12B	13/28	8/22
+ field guide	20/28	16/22

I stopped tuning the list after one revision, so I wasn't just fitting it to my own test photos.

Fix 2: fine-tuning on a DGX Spark (and why it lost)

I also had access to an NVIDIA DGX Spark, so I tried the "proper" fix too: a LoRA fine-tune of Gemma 4 12B on 1,193 photos from DIMPSAR, an open dataset of Indian plant photos from Karnataka and Kerala (Pushpa B.R. & N. Shobha Rani, Data in Brief 2023, CC BY 4.0).

To keep the app's output format, each training answer was Gemma's own answer with only the species name corrected. I trained a LoRA on the language layers (rank 16, one epoch, about 70 minutes), merged it, and patched the weights into Ollama's official gemma4:12b file so it kept working with images.

(measured on the DGX)	Stock	Fine-tuned
28 test photos	12/28	13/28
DIMPSAR validation	16/100	20/100
Confidence	0.6–0.95	1.0 on every photo

+1 photo, and it became 100% sure of everything, which would switch off the app's "second look" and Mystery finds. Two lessons I'm taking with me:

Check the scale of numbers you copy into training data. 4% of Gemma's own answers said "confidence": 85 instead of 0.85. The app had quietly clamped those, so nobody noticed until the fine-tuned model said 1.0 for everything.
In structured output, the loss is mostly the long text. Validation loss fell from 1.30 to 0.33, but name accuracy only went from 16% to 20%. The species name is a few tokens next to four descriptive sentences.

So the app ships with stock Gemma 4 plus the field guide. A 107-line text file beat 70 minutes on a DGX Spark.

Other models I measured

On my first 13 test photos: Gemma 4 12B 10/13, Qwen3.5 9B 9/13, MiniCPM-V 4.5 8/13, Qwen2.5-VL 3B 8/13 (it said "100% sure" about almost everything), and Gemma 4 12B QAT on llama.cpp 7/13. That last one was faster, but adding Gemma 4's new MTP drafter gave no speed-up on a 6 GB GPU, because when the model is split between GPU and CPU, checking the draft costs as much as it saves. Every model did fine on clean internet photos. Real, messy garden photos are where Gemma pulled ahead.

What still doesn't work

Some plants are still wrong: moringa (called curry leaf; both have small leaflets), a ZZ plant, an Indian almond tree and one rose plant. That's why the app always shows "could also be", Mystery finds and the ✏️ correction.
Birds and insects: the field guide doesn't cover them yet. My one bird test is still a miss.
Its confidence is a hint, not a measurement. It says 85% for right and wrong answers alike.
My test labels are careful judgements, not a botanist's. A couple of photos I left out of the score because I couldn't identify them myself.

I built this with Claude Code as my pair programmer, including the scoring scripts that made every decision above a measured one.

Why Does Open Innovation Matter?

For this project, open isn't a nice-to-have. Every fix above depended on it.

It works where there's no signal. Gardens and parks are where mobile data is weakest. The phone only needs a local link to the laptop.
My photos stay mine. Photos of my street and garden, with times attached, are location data. They never leave my phone and laptop.
It costs nothing per photo. I ran hundreds of test photos through Gemma while building this, with no API bill and no rate limit. A closed API would have made that kind of measuring expensive.
I could look inside and choose the fix. With an open model I could try prompts, swap models, quantize, add an MTP drafter, and fine-tune the weights myself, then measure which actually worked. The answer (a text file beat a fine-tune) is only something you can learn when you're allowed to try both.
Anyone can make it local. field_guide.csv is a plain file. Someone in Kerala, Kenya or Kansas can replace it with their own region's plants, and the same open model becomes their field guide. That's the part I'm most excited about: open models plus open, editable local knowledge.

Prize Categories

Best Use of Gemma. Gemma 4 12B (open weights) runs locally through Ollama and does all of the vision identification, the second-look reasoning, the quests and the walk journal. I also fine-tuned it (LoRA on a DGX Spark) and tested its MTP drafter and QAT version along the way.

Now close the laptop and go and see what's growing by your gate. 🌿

We made SigNoz's LLM observability actually reachable…

Gajanand V Dhayagode — Sun, 26 Jul 2026 17:01:23 +0000

-and added the one signal it was missing

Track 01 — AI & Agent Observability. A hackathon project built from scratch, July 20–26, 2026.

There's a sentence we said to each other on day one and then kept saying all week, because it kept being true:

An API 200 is not evidence that a feature works.

This post is the story of a project that lives entirely inside that sentence. It's about a feature SigNoz already shipped that almost nobody can reach, a real bug that makes part of it impossible, and a missing signal — measured answer quality — that we built on top. And it's about how we used SigNoz not as a black box but as the thing under test.

The surprising starting point: SigNoz already built the machinery

Most "add observability to LLMs" projects start from zero. Ours didn't, and that's the interesting part.

SigNoz v0.133.0 already ships a complete LLM-observability product in open source: a dedicated frontend module, two custom OpenTelemetry collector processors (an LLM-pricing processor and a span-mapper processor), and CRUD APIs behind them. The machinery to ingest gen_ai.* spans, price them, normalize vendor-specific shapes, and render them on a dashboard is there, in the OSS repo, today.

So our thesis wasn't "SigNoz is missing LLM observability." It was sharper and, we think, more useful:

SigNoz shipped the machinery. We make the shipped feature actually reachable — and add the one signal it's missing: measured answer quality.

That framing mattered to us ethically, too. Nothing in this project reads "SigNoz is broken." It reads: they built something good, three defaults and one bug stand between a self-hoster and it, and here's the fix — upstreamed.

The three silent gates

Here's what actually stands between a fresh docker compose up and a working LLM-observability feature. Every one of them fails silently — the backend returns success while the frontend hides the result, and nothing tells you why.

Gate 1 — `enable_ai_observability` is default-disabled

The whole feature sits behind an experimental flag that ships off. With it off, the UI route for LLM observability silently redirects to /home. No error, no "coming soon," no toast. You click the thing and you're just... somewhere else. At the time we hit it, there were zero web results for the flag name.

Gate 2 — `use_dashboard_v2` is default-disabled

The seeded LLM dashboards are v2 dashboards. Import one over the API with this flag off and it succeeds — 200 OK — and then renders as an empty "Welcome to your new dashboard" shell. The data is flowing. The panels are defined. The screen is blank.

This is where the day-one sentence was born. Both gates let the backend succeed while the frontend hides the result. We got bitten by both, in one afternoon, before we understood the pattern.

The fix for gates 1 and 2 is to ship both flags together — and it has to be both:

SIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY=true
SIGNOZ_FLAGGER_CONFIG_BOOLEAN_USE__DASHBOARD__V2=true

Setting only one still gives you an empty shell — the exact failure we're fixing. So any "dashboards-as-code" deliverable that sets one flag is quietly reproducing the bug.

Gate 3 — the registries ship empty

Two registries back the feature — llm_pricing_rules (model prices) and span_mapper_groups (vendor→spec mapping presets). Both ship with zero rows. No seeded prices, no mapping presets. And there's a nasty second-order trap here: SigNoz's own CrewAI docs tell users to instrument with OpenInference — which emits zero gen_ai.* attributes — so a user who follows the docs faithfully lands on a seeded dashboard that shows nothing, and has no way to know the mapping layer is what's missing.

The one real defect: cost is impossible regardless of configuration

The three gates are configuration. This one is a bug.

SigNoz's LLM-pricing processor and span-mapper processor are generated into the collector config correctly — we verified the rendered YAML. But neither of them is ever wired into a pipeline:

GenerateCollectorConfigWithLLMPricingProcessor and GenerateCollectorConfigWithSpanMapperProcessor both write only collectorConf["processors"].
Neither touches service.pipelines.traces.processors.

In OpenTelemetry Collector terms, that's dead config: a processor that exists but is in no pipeline never runs. So even with the flag on and a correctly-rendered gpt-4o pricing rule, the effective traces pipeline was still [signozspanmetrics/delta, batch], and ClickHouse showed no cost attributes — not _signoz.gen_ai.total_cost, not _signoz.gen_ai.cost_input / cost_output (those are the real emitted names; there is no gen_ai.usage.cost output attribute).

The good news: the codebase already does this right somewhere. The logs feature wires its generated processor straight into the pipeline. So the fix is small, it follows in-repo prior art, and it's the kind of change a maintainer can verify in two minutes:

enable enable_ai_observability → add a pricing rule → observe the effective traces pipeline is still [signozspanmetrics/delta, batch] → cost attributes never appear.

We filed it upstream as SigNoz/signoz#12268 — a small, focused change that cites the prior-art file and ships a clean reproduction. A teammate's fix, not a hackathon drive-by.

The part we're proudest of: measuring whether the answers are any good

Everything above makes the shipped feature reachable. But there's a signal SigNoz's LLM product genuinely does not have, and it's the one that matters most for agents in production: is the model's answer any good, and did the last release make it worse?

Cost, latency, and token counts tell you the model ran. They say nothing about whether it was right. So we built an eval layer.

Emitting a signal SigNoz has never seen

OpenTelemetry specs an evaluation signal — gen_ai.evaluation.result (name, score, label, explanation). It has zero references in the SigNoz codebase. We emit it: an LLM-as-judge scores an answer, and the score travels into SigNoz over OTLP as spec-shaped telemetry.

Crucially, how you carry it decides whether it's usable. We learned this the hard way and built a probe (probe_eval_carriers.py) that emits the same score three ways side by side to prove which one survives:

Span events are stored in ClickHouse as Array(String) — you cannot filter or aggregate inside them. An eval you can't GROUP BY is not auditable.
A spec-compliant log record and a dedicated evaluation span both land as queryable attributes.

So we carry every score on the two that SigNoz can actually aggregate, never on a span event.

The judge earns its place by benchmark, not reputation

The judge is the load-bearing piece: every downstream claim is only as trustworthy as the score it produces. So we picked it by measuring it against our actual task, and we picked it to run locally on open weights — gpt-oss:20b on Ollama, free and self-hostable.

That last choice is architecture, not thrift. This entire project exists to remove a gate between a self-hoster and a feature. A judge that needed a paid API key would reintroduce exactly that kind of gate — now you need a billing account before you can evaluate anything. A free, self-hostable judge works for the same person we're unblocking.

We benchmarked five candidates. The results were not what "bigger is better" would predict:

model	valid JSON	grounded	fabricated	separation	jitter (temp 0)	p50 latency
gpt-oss:20b	3/3	1.000	0.000	+1.000	0.000	3.16s
gpt-oss:120b	3/3	1.000	0.133	+0.867	0.250	4.66s
nemotron-3-nano:30b	flaky	1.000	0.270	+0.730	0.180	5.38s
deepseek-v4-flash	—	HTTP 403 (not in tier)
gemma4:31b	—	returned non-JSON

The bigger model was the noisier one. On a repeated fabricated input at temperature 0, gpt-oss:120b returned 0.25 / 0.00 / 0.15 — a 0.25 spread — while gpt-oss:20b returned 0.00 every single time. A quality-regression alert lives or dies on that stability: a judge that wobbles ±0.25 makes "quality dropped 12%" indistinguishable from measurement noise, and can name the wrong release as the culprit. We chose the model our alert can actually trust, and it happened to be the smaller, faster, free one.

Every score also carries the two attributes that make it reproducible after the fact:

gen_ai.evaluation.judge.model — which model scored it
judge.prompt.version — which rubric version scored it

So "which judge and which rubric produced this score?" is a GROUP BY, not a shrug.

The alert that names the release that regressed

The payoff is a threshold alert: avg(gen_ai.evaluation.score.value) below 0.7, over a 30-minute window, grouped by service.version. When a release's answer quality slides under the floor, the alert fires and names the release. No anomaly detection — that's enterprise-gated in SigNoz — just a threshold on a signal we made aggregatable.

We measured the full loop end to end against a live instance, not estimated it:

emit at 16:21:43Z → alert transitioned to firing at 16:25:40Z
span-to-alert = 238s (~4 min)

And here's the part that matters for anyone building on this: that time is almost entirely SigNoz's own machinery — the 120s eval_delay plus the 1-minute evaluation interval. The judge adds only ~2.5s per call, and because spans are timestamped when the run starts, even that overlaps the eval_delay rather than stacking on it. The eval layer adds essentially nothing to detection time; SigNoz's evaluation interval is the floor.

For the demo we use a deliberately loud drop — measured mean 0.958 → 0.167 — so the mechanism is legible in one glance. That's a demo choice, not a limitation: the alert fires on the same 0.7 floor whether the slide is a cliff or a subtle crossing.

How we deployed it — and made it reproducible

We ran SigNoz self-hosted in WSL2 (Ubuntu 24.04) on native Docker Engine — not Docker Desktop, whose ClickHouse Keeper is known to segfault. Deployment is via SigNoz Foundry (foundryctl), their "one config, one command" tool: you declare the whole deployment in a casting.yaml, forge expands it into platform manifests plus a checksum-pinned casting.yaml.lock, and cast deploys it.

Because the judges of this hackathon may re-run Foundry to reproduce our deployment, we didn't hand-edit generated files — we put the fix in the source of truth. Our casting.yaml sets both gates as first-class env overrides on the SigNoz service:

spec:
  deployment:
    flavor: compose
    mode: docker
  signoz:
    spec:
      version: v0.133.0
      image: signoz/signoz:v0.133.0
      env:
        SIGNOZ_FLAGGER_CONFIG_BOOLEAN_ENABLE__AI__OBSERVABILITY: "true"
        SIGNOZ_FLAGGER_CONFIG_BOOLEAN_USE__DASHBOARD__V2: "true"

We verified this is the right way to do it: forge merges those overrides into the defaults (it doesn't wipe the DB/store env), the flags render into the generated compose.yaml, and — critically — forge is deterministic: re-running it produces a byte-identical lock. So a judge who clones the repo and runs foundryctl forge && foundryctl cast gets the feature-reachable deployment, not the default empty shell. The casting file is part of the argument, not a checkbox.

What we'd tell the next person

Treat a 200 as the beginning of verification, not the end. Both gates that cost us the most time were backends succeeding while frontends hid the result. Visually confirm every dashboard; never trust the POST.
Pick your LLM judge by measuring it, not by its size or name. The stability of the judge is the stability of your alert. Bigger lost.
Choose carriers your backend can aggregate. A beautiful eval signal stuffed into a span event is invisible to GROUP BY. Know your storage model before you emit.
Put the fix in the source of truth, not the generated output — so reproduction reproduces the fixed thing.

SigNoz built a genuinely good LLM-observability product. We made it reachable, filed the bug that was blocking cost, and added the measured-quality signal it was missing — all in OSS, all reproducible from one casting.yaml.

Repo (with casting.yaml + casting.yaml.lock for reproduction): https://github.com/gajanand27-05/SigNoz-conform

DEV Community: Gajanand V Dhayagode

My fine-tuned Gemma blamed a critical haemoglobin on "how the sample was handled". My eval gave it a pass.

What I Built

Demo

Code

gajanand27-05 / report-saathi

Report Saathi: understand your lab report

How I Built It

1. The reader has to prove every number

2. The rules: a table a doctor can check

3. The explainer, and the excuse it learned

One deploy surprise

What still doesn't work

Why Does Open Innovation Matter?

Prize Categories

My AI field guide called castor a "Virginia Creeper". A 107-line list fixed more than fine-tuning did.

What I Built

Demo

Code

gajanand27-05 / pocket-naturalist

Offline field guide for walks: snap a plant or bird, local Gemma 4 identifies it and sends you on the next quest. Hacktoberfest 2026 Touch Grass.

🌿 Pocket Naturalist

How it works

How I Built It

Day 1: making it honest

Day 2: the real walk, and a bias I didn't expect

Fix 1: a field guide in plain text

Fix 2: fine-tuning on a DGX Spark (and why it lost)

Other models I measured

What still doesn't work

Why Does Open Innovation Matter?

Prize Categories

We made SigNoz's LLM observability actually reachable…

The surprising starting point: SigNoz already built the machinery

The three silent gates

Gate 1 — enable_ai_observability is default-disabled

Gate 2 — use_dashboard_v2 is default-disabled

Gate 3 — the registries ship empty

The one real defect: cost is impossible regardless of configuration

The part we're proudest of: measuring whether the answers are any good

Emitting a signal SigNoz has never seen

The judge earns its place by benchmark, not reputation

The alert that names the release that regressed

How we deployed it — and made it reproducible

What we'd tell the next person

Gate 1 — `enable_ai_observability` is default-disabled

Gate 2 — `use_dashboard_v2` is default-disabled