ur rag can return 200 and still be cooked — how we built Goose on SigNoz

Karthikeya Atthavelli — Sat, 25 Jul 2026 22:47:44 +0000

ok so real talk. your latency graph looks clean. error rate is chilling. every /chat is HTTP 200.

and the answers are just… wrong. like confidently wrong. mid.

that’s the thing that lowkey haunted us after HackRx RAG 6.0. like ~1000 teams, we somehow landed top 30, we were deep in evals / retrieval / “is this grounded tho”, and the whole time our “monitoring” was basically vibes + batch scripts. RED metrics (rate, errors, duration) stay green while the model yoinks the wrong KB. for a startup shipping RAG that gap is actually lethal cuz users bounce before your SLO even flinches.

so for Agents of SigNoz we built Goose — catch silent AI quality fails, put them in SigNoz, alert on the cliff, then an MCP agent writes an RCA and can auto-fix the break. this post is what we actually did / hit, not a docs remix. you’ll see the signals, the gotchas, and a real e2e run with numbers.

the failure mode nobody puts on the dashboard

normal API: wrong = 5xx or timeout. rag chatbot: wrong = success arc.

retriever returns something
llm writes a smooth answer
handler returns 200
span status: OK (fr)

but asked category ≠ queried category. user asks a custom fact, pipeline silently pulls billing docs. or tool output is empty/invalid and the model just freestyles from “general knowledge”.

on hackrx that silent miss is the diff between “cool demo” and “i would never ship this”. we needed signals for is this answer actually grounded / routed right — not just is the process alive.

what Goose even is

short version:

demo rag emits otel quality junk every turn
signoz eats traces/metrics/logs and shows the cliff
alert when quality.score tanks
agent hits signoz mcp, pulls evidence, writes markdown rca
optional auto-remediate (POST /admin/fix-retriever for our injected break)

track is AI & Agent Observability.

demo flow we used: run ~100 custom-fact questions healthy (answers deliberately weird — “we live on Mars” — so you can tell its grounded in our chroma store not wikipedia). then break retrieval. same questions. still 200s. scores go brrr into the floor. answers flip. signoz shows the cliff. agent investigates + can fix.

how we actually used SigNoz

1. otlp everything into one place

rag app ships otlp into signoz. each chat turn ain’t just a request span — we slap quality attrs + metrics on it:

quality.score — the main cliff metric
quality.entity_match — asked category match retrieved?
tool.output_valid — was retriever/tool output even usable
gen_ai.* + zone attrs (asked / queried)
structured quality logs for the same mess

silent fails are cross-signal by nature. trace can look fine (OK) while the metrics panel shows quality.score falling like 1.0 → 0.2. in signoz we open the dashboard cliff then jump a span that still says OK but tool.output_valid=false. that combo is the whole thesis.

drop your dashboard screenshot here — quality.score healthy then cliff

2. dashboard for the cliff, not vanity charts

we didn’t want another “cpu go up” board. ours is basically:

latest / avg quality.score
entity_match / misroute vibes
tool.output_valid rate
traffic still healthy while quality dies (thats the joke)

if volume stays up and quality dies, you know its not “the box is down”, its “the answers got dumb”.

screenshot: healthy avg ~1.0 vs broken ~0.2

3. alert = answers got dumb, not cpu spicy

alert: quality.score below 0.6. channel = webhook into our agent on :8020/webhook/alert.

this is the bit that made it feel real. signoz isn’t just a museum of charts. its the pager for quality. same muscle memory as a classic outage page, except the outage is semantic.

screenshot: alert rule / fired alert

4. mcp agent as the on-call that reads telemetry

alert fires → goose doesn’t invent lore. it talks to signoz mcp, grabs recent quality evidence, dumps a markdown rca (what tanked, misroute hints like asked vs queried zones, what to do).

for the hackathon remediator we call POST /admin/fix-retriever when the report / alert value smells like our injected misroute. irl you’d swap that for reindex / model rollback / feature flag whatever — pattern stays: signoz evidence → agent decides → action.

also being honest: auto-fix does not magically fix “we changed embeddings and everything drifted”. our remediator is for the routing break we demo. don’t overclaim that part, judges can smell it.

screenshot: agent report markdown

e2e we actually ran (not vibes)

healthy traffic → break → webhook → fix. autoheal off on purpose so the agent had to be the one that healed it.

healthy (~5 min, 25 q) — avg score 1.0, retriever not broken
broken (5 q) — scores 0.2, tool.output_valid=false, answers drift to earth/general knowledge etc
agent — webhook → mcp investigate → POST /admin/fix-retriever ok
verify — mars answer back, score 1.0, retriever healthy

this is literally the loop we wished we had during hackrx crunch. not only “did the batch score drop in a csv” but “did prod quality just cliff and can something investigate without me clicking around for 40 mins”.

instrumentation (the part you can copy)

every turn we roughly do this then export otlp:

span.set_attribute("quality.score", quality_score)
span.set_attribute("quality.entity_match", 1.0 if asked == queried else 0.0)
span.set_attribute("tool.output_valid", tool_output_valid)
span.set_attribute("quality.asked_zone", asked)
span.set_attribute("quality.queried_zone", queried)

quality_score_gauge.set(quality_score)

point OTEL_EXPORTER_OTLP_ENDPOINT at signoz otlp.

gotcha we ate so you don’t have to: if you start the rag before signoz is up, otel can basically no-op / disable and then your dashboards look haunted (empty) even tho the app “works”. restart rag after signoz is healthy. we wasted time on this. not proud.

another dumb one: localhost vs 127.0.0.1 on mac — ipv6 being spicy — our e2e died with connection refused until we pinned 127.0.0.1. tiny, but its the kind of detail you only get from actually running the loop at 3am.

what worked / what didn’t

worked

quality metrics next to traces in one ui (signoz) = way less context switch brain death
alerting on quality.score feels correct. cpu alerts would’ve slept through this incident
mcp for rca is actually useful when the evidence is already in signoz, not in a random spreadsheet

didn’t / don’t lie about it

in-app autoheal can steal the demo if you forget to turn it off (we did, then fixed)
one remediator ≠ every root cause. embedding change ≠ misroute break
starting services in the wrong order will gaslight you into thinking instrumentation is broken

what i’d tell hackrx-me: instrument semantic quality on day 1. batch evals are cute. production still needs a cliff on a graph.

wrap

goose exists cuz top 30 at hackrx still left a hole — strong rag, blind when it fails quiet. for this hackathon we closed it: quality signals in, dashboards + alerts in signoz, mcp agent out with rca (+ optional fix). we’re using the same mindset for our startup rag stuff too, because “http 200 but wrong” is not a hypothetical, its the default failure mode.

one question for your stack: when the answer is wrong but http is 200, what breaks on the graph?

if the answer is “nothing”… yeah. that’s the bug.

DEV Community: Karthikeya Atthavelli