DEV Community: Abhishek Gupta

I Built a Trace Funnel Over My AI Agent. The Conversion Rate Came Back NaN.

Abhishek Gupta — Sat, 18 Jul 2026 17:26:23 +0000

I asked SigNoz a simple question: of the runs where my agent planned an answer and called its search tool, how many reached the LLM step? The funnel said zero, for a model that had demonstrably run in ten traces. Chasing that zero led somewhere better than the answer I wanted: an analytics endpoint that returns HTTP 500 where "0%" belongs, and a one-line guard in the SQL that explains exactly why.

This is a warm-up post for the Agents of SigNoz hackathon. I self-hosted SigNoz v0.132.2 with Foundry (the docs walk through it), pointed a small AI agent at it, and spent a few days on a feature I hadn't seen anywhere else. My first warm-up post covered trace-to-log correlation; this one is a different corner of the product.

Why funnels, and why over agent traces

Conversion funnels are an old product-analytics idea: land, add to cart, check out, measure the drop-off at each step. SigNoz took that shape and pointed it at traces. You define an ordered list of span-name steps, and it tells you what fraction of traces made it through each one. I went hunting for this in the LLM-observability tools I already knew, and came up empty. Langfuse, LangSmith, Phoenix, Braintrust: plenty of tracing, but I didn't find a conversion funnel over trace steps in any of them.

For an agent, that shape is exactly right. An agent run is a small pipeline. Plan, call a tool, call the model, answer. If some fraction of runs never reach the model step, or the tool step silently drops out after a deploy, a funnel is the cleanest way to see it. So I opened Traces, found the Funnels section, and started building.

The data I was working with

A single session of agent runs left 22 traces under one service, warmup-agent. Small n, but as it turns out, more than enough to hit the edge. Every trace has an agent answer span and a tool search_tool span. Then there's the LLM step, and this is where it gets interesting. I switched models a few times during that session, and my spans are named following the OpenTelemetry GenAI semantic conventions (still marked experimental), which specify the span name as {operation} {model}. One logical operation, but the name changes with the model.

I confirmed the shape straight from ClickHouse:

docker exec signoz-telemetrystore-clickhouse-0-0 clickhouse-client -q "SELECT name, count() AS spans, count(DISTINCT trace_id) AS traces FROM signoz_traces.distributed_signoz_index_v3 WHERE name LIKE 'chat %' GROUP BY name ORDER BY spans DESC FORMAT PrettyCompactMonoBlock"

   ┌─name───────────────────────┬─spans─┬─traces─┐
1. │ chat gpt-4o-mini           │    20 │     10 │
2. │ chat gemini-3.1-flash-lite │    16 │      8 │
3. │ chat gpt-oss-120b          │     8 │      4 │
   └────────────────────────────┴───────┴────────┘

That's 44 chat spans across 22 traces, split three ways by model. The agent answer and tool search_tool spans match all 22 traces each; the LLM step matches at most 10 of them under any single name. This is correct instrumentation. The convention says the model belongs in the span name, my exporter obeys, and the result is that "the LLM call" has no single name to funnel on.

Here's the naming, straight from my agent:

with tracer.start_as_current_span(f"chat {MODEL}") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", MODEL)

Building the funnel

My first funnel had three steps in the order I think of my agent: agent answer, then tool search_tool, then chat gpt-4o-mini.

Steps one and two matched 22 out of 22. Step three matched zero.

Zero. For a model that demonstrably ran in ten traces. At first I assumed I'd broken my own instrumentation. That's the honest reflex: when a tool shows zeros on your data, you suspect your data. I went back to my exporter config, re-checked the span names against ClickHouse, even built a one-step funnel containing only chat gpt-4o-mini. That one matched 10 traces. So the spans exist, the names are right, and the funnel still says zero when the step sits third.

The explanation took me longer than I'd like to admit: funnels enforce strict temporal ordering within each trace, taking the first matching span per step, and my agent calls the LLM to plan before it calls the tool. The first chat span in every trace fires before tool search_tool, so no trace satisfies "chat after tool," and step three legitimately converts to zero. Fair enough, that's what a funnel means. But notice what the number can't tell you: a step reading 0 looks identical whether the model never ran or ran a thousand times in a different order.

Reordering the steps to agent answer → chat gpt-4o-mini → tool search_tool gave me a real conversion rate: 9.09%.

And that's before the span-name fragmentation even enters. Under any single name, the LLM step can only ever match the traces of that one model, at most 10 of my 22. Now imagine the realistic production event: a fallback or a version bump quietly retires a model, its span name lingers in your funnel definition, and the step's match count goes to zero for a second reason. That's where it stopped being a quirk and became a crash.

The crash

With the third step pointed at a model that had emitted no spans at all, the funnel page went to "No data" and the conversion rate read 0.00%. The API underneath was doing something worse. The per-step endpoint (/api/v1/trace-funnels/analytics/steps) still answered HTTP 200, honestly reporting total_s3_spans: 0. But both overview endpoints, the ones that compute the conversion rate, returned HTTP 500 with this body:

app.ApiResponse.Data: []*v3.Row: v3.Row.Data: unsupported value: NaN

A step that matches nothing should be a zero-percent conversion. Zero is a perfectly good answer to "how many made it this far." Instead a zero-converting final step produces a NaN the response can't carry, the request dies with a 500, and the UI papers over it as "No data." I re-verified all of this today against my live instance; the exact curl commands, request bodies, and responses are in the repo if you want to reproduce it end to end (creating funnels needs an editor or admin token; a read-only key gets a 403).

So I went reading the query builders that generate this SQL, and the shape of it is almost funny. The full-funnel query already guards its conversion division:

round(if(total_s1_spans > 0, total_sN_spans * 100.0 / total_s1_spans, 0), 2) AS conversion_rate

That guard is why a zero-converting funnel still reports a sane 0% for conversion. But sitting in the same SELECT are two aggregates with no such protection:

avgIf(..., <full funnel completed>)          AS avg_duration,
quantileIf(0.99)(..., <full funnel completed>) AS latency

When no trace completes the funnel, those aggregate over an empty set, and the response carries a value JSON can't encode. The step-transition query has the same two unguarded aggregates, and its conversion division is missing the guard its sibling already has. The fix that already exists one function away is the fix the other one needs. I filed it as SigNoz#12143 with this repro and offered a PR.

What matters for anyone using funnels today: a zero-converting step is a live crash behind a quiet UI, and GenAI span naming plus ordering makes zero-converting steps very easy to create by accident.

What actually works, and one thing that doesn't

The fix that held up: wrap the LLM call in a stable parent span. If every model call lives inside an outer llm step span with a constant name, you funnel on llm step and the model name stops mattering to the funnel. The GenAI-named child spans stay exactly as the convention wants them, so you lose nothing.

The tempting fix that doesn't work: the step filter clause. Funnel steps do accept an additional filter, and my first thought was to match on a stable attribute like gen_ai.operation.name = "chat" instead of the name. But the filter narrows within a span-name match rather than replacing it. A step still keys on one literal name, so no attribute filter can stitch chat gpt-4o-mini and chat gemini-3.1-flash-lite back into one step. Filters are still useful inside a step (scoping to one environment, say), just not for this. Knowing that saves you the hour I would have spent on it.

Two operational habits, both learned the slow way. Before trusting any multi-step funnel, sanity-check each step as a single-step funnel first; that's a fast, honest span count per name, and it's how I proved my zeros were an ordering problem rather than missing data. And order your steps by when the spans actually start inside a trace, not by your mental model of the pipeline. My mental model said the LLM answers last; my traces said the LLM plans first. The funnel believed the traces.

If you hit a zero of your own, this is the whole diagnosis in one picture:

What I took away

Trace Funnels are, for my money, one of the most underrated primitives in SigNoz for agent work. Measuring drop-off across the steps of a trace maps onto agent pipelines so naturally that I'm surprised I couldn't find it in the dedicated LLM tools. The sharp edges are specific and avoidable once you can name them: steps match literal span names, GenAI naming guarantees your most important step won't have one, ordering follows the trace rather than your intentions, and a step that converts to zero crashes the overview API instead of reporting 0%. Funnel on a wrapper span with a name you control, and check your ordering with single-step funnels first.

Rough corners are normal in a young feature, and the useful response is a repro rather than a complaint. Two days with this thing and I'm more convinced it belongs in an agent developer's toolkit, not less.

Resources

The bug: SigNoz#12143, with the repro from this post
Full request/response log and copy-pasteable curls: funnel-api-repro.md
OpenTelemetry GenAI span conventions: gen-ai-spans.md
SigNoz docs · install path: SigNoz on Docker

AI assistance disclosure. I used Claude as a research assistant and as an editor for structuring and tightening this post. The exploration, the self-hosted instance, the data, and the screenshots are mine. Every number was verified against my live ClickHouse and my own funnel analytics API on July 17–18, 2026; the full request/response log is in the repo.

My AI Agent Took 21 Seconds to Answer "What Is 2 + 2" — SigNoz Found Out Why in Two Clicks

Abhishek Gupta — Sun, 12 Jul 2026 18:28:38 +0000

Warming up for the Agents of SigNoz hackathon, I self-hosted SigNoz, pointed a tiny AI agent at it, and fell hard for one feature: the one-click jump from a slow trace span into the exact logs that explain it. (2 + 2 is not supposed to be a load-bearing operation.)

TL;DR — My agent answered "what is 2 + 2" in 20.9 seconds. Its own logs, read in isolation, looked perfectly healthy. In SigNoz, the trace flamegraph showed one span eating 18.31 s — 87.62% of the request — and clicking that span's Logs tab exposed a silent retry loop that never once surfaced as an error. Two clicks from symptom to root cause. That trace↔logs correlation is my favorite feature, and this post is the story of finding it. (Then I pointed SigNoz's MCP server at the same data and had an AI assistant re-run the whole investigation — down to the exact line number of the bug.)

Why a warm-up post about an AI agent

The hackathon's pre-event challenge — Warm Up Before You Build — asks you to self-host SigNoz, send it real telemetry, explore the product, and write about your favorite feature.

I could have pointed it at a hello-world web app. Everyone does. Nobody remembers the hello-world blog. But the main event is themed agent observability, and agents have a nasty property that makes them the perfect stress test for an observability tool: they fail quietly. No 500s, no stack traces. They loop, they retry, they burn tokens on tangents — and then return a perfectly plausible answer. The failure isn't a crash; it's behavior. So my warm-up plan was simple: build a deliberately imperfect agent, wire it to SigNoz, and see how fast I could catch it misbehaving.

Spoiler: faster than I expected. (Narrator: it was not, at first, faster than expected. The narrator was being generous, the same way your agent's logs are about to be generous with the truth.)

Self-hosting SigNoz (~15 minutes, one gotcha)

SigNoz's current install path is Foundry, its new installer CLI. The old flow is gone, and the docs are blunt about it: the legacy install.sh script and the bundled deploy/ Compose files are "deprecated as of SigNoz v0.130.0 and are no longer maintained or distributed". If you find a 2025 tutorial telling you to git clone && docker compose up, close the tab.

Prerequisites: Docker, and a shell that can run the install script. That's genuinely it — no cloud account, no signup, no ingestion key, no sales call asking if I have "fifteen minutes to talk about my observability journey." Then three steps:

# 1. Install the foundryctl CLI
curl -fsSL https://signoz.io/foundry.sh | bash

# 2. casting.yaml — tell Foundry to deploy via Docker Compose
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker

# 3. Cast it
foundryctl cast -f casting.yaml

A couple of minutes later, docker ps showed the whole stack healthy — the SigNoz server, an OTel collector (the "ingester"), ClickHouse for storage, ClickHouse Keeper, and a Postgres metastore. UI on http://localhost:8080, OTLP ingestion on 4317 (gRPC) and 4318 (HTTP). Every byte of telemetry stays on my machine. (I'm on Windows 11 — Docker Desktop with the WSL2 backend, and the install script runs fine from Git Bash.)

The gotcha worth knowing: my first OTLP exports were rejected with connection resets, and the ingester logs showed the collector stuck in an OpAMP error/restart loop. The cause was hilariously mundane — I hadn't created the admin account yet. SigNoz's collector gets its config via OpAMP from the server, and until the first user/org exists it has nothing to serve. The moment I signed up at localhost:8080, the collector settled and ingestion turned green. If your self-hosted collector seems broken on first boot: create your account first. (It is, once again, always the auth. It has never once not been the auth.)

The Services page a few minutes later: one application, warmup-agent, P99 latency 21,818 ms — derived automatically from its spans, no metrics code written. For "2 + 2". Most pocket calculators clear that in under a second, batteries included. We'll get to that.

The agent, and how it's instrumented

My agent is a single Python file doing the classic loop real frameworks do — plan with an LLM (Gemini's gemini-3.1-flash-lite, via its OpenAI-compatible endpoint), call a tool, answer with an LLM:

def answer(question: str) -> str:
    plan  = llm("plan", f"Plan how to answer: {question}")   # LLM call #1
    facts = search_tool(plan)                                # tool call
    return llm("answer", f"Answer {question} using: {facts}")# LLM call #2

For instrumentation I went OTel-native and hand-rolled spans following the OpenTelemetry GenAI semantic conventions, the vendor-neutral standard (gen_ai.* attributes) for recording model names, token usage, agent steps, and tool calls. One thing that cost me ten minutes of confused searching: these conventions moved out of the main semantic-conventions repo into their own home at open-telemetry/semantic-conventions-genai, and the old opentelemetry.io/docs/specs/semconv/gen-ai/ URL is now just a "this has moved" stub. The two pages you actually want are gen-ai-spans.md for model calls and gen-ai-agent-spans.md for agents and tool execution. They're still experimental, so expect attribute names to shift under you.

Every LLM call becomes a span carrying its contract with the outside world:

with tracer.start_as_current_span(f"chat {MODEL}") as span:
    span.set_attribute("gen_ai.operation.name", "chat")
    span.set_attribute("gen_ai.request.model", MODEL)
    span.set_attribute("gen_ai.agent.name", "warmup-agent")
    # ... make the call ...
    span.set_attribute("gen_ai.usage.input_tokens", in_tok)
    span.set_attribute("gen_ai.usage.output_tokens", out_tok)

Two exporters point at my local SigNoz — OTLPSpanExporter for traces and OTLPLogExporter behind a stock logging handler for logs — both to http://localhost:4318. That logging handler is the quiet hero of this story: it stamps every log record emitted inside a span with that span's trace_id and span_id. Remember that detail.

(If you'd rather not write spans by hand: opentelemetry-instrument plus the GenAI instrumentation packages will auto-capture the same gen_ai.* attributes for OpenAI, LangChain, and friends with zero code changes. I wanted to feel the semantic conventions under my fingers at least once.)

The mystery: 21 seconds for "2 + 2"

First real runs:

run 1: arithmetic: 2 + 2 = 4 (source: math)  (21.9s)
run 2: Arithmetic: 2 + 2 = 4 (source: math)  (20.6s)
run 3: arithmetic: 2 + 2 = 4 (source: math)  (20.9s)

Correct answer. Absurd latency.

My first guess was the lazy one: free-tier Gemini being free-tier Gemini. I'd already fought its rate limits earlier that day (the retry delay in my run loop exists for a reason), so a slow API felt plausible enough that I almost didn't investigate. Step one of the Five Stages of Ignoring a Bug is always "blame the free tier." It is never the free tier's fault quite as often as we'd like it to be.

Two things stopped me. Twenty seconds is slow even for a free tier having a bad day. And scrolling my terminal logs, nothing looked wrong — no errors, no exceptions, a clean INFO trail, the software equivalent of a suspect saying "no further questions, officer" with total confidence. This is the agent failure mode in miniature: everything "works," and something is deeply off.

Time to look at what SigNoz saw.

Reading the flamegraph: where did 19.5 seconds go?

I opened the Traces Explorer, and rather than scrolling, made the query builder do the work — one filter, durationNano >= 15s, and only the pathological runs remained. Clicked one.

One glance killed my Gemini theory: the root agent answer span is 20.89 s, and sitting under it is a monstrous tool search_tool bar — 18.31 s, 87.62% of total execution time. The two chat gemini-3.1-flash-lite spans flanking it are slivers (1.57 s for planning, a blink for the answer). The LLM I'd been ready to blame was never the problem; my tool was. The span's attributes (that's the GenAI conventions paying off) confirmed it: gen_ai.tool.name: "search_tool", and — suspicious — tool.attempts: 3.

It's called a flamegraph. I did not expect that name to be quite this on the nose — 87.62% of my request was, functionally, on fire, and my terminal logs were standing next to it holding a cup of coffee insisting everything was fine:

A trace tells you where the time went. It doesn't tell you why. Historically, "why" is where debugging gets miserable: copy the timestamp, switch to your logging tool, paste an approximate time window, squint at clock skew, guess which lines belong to this exact request.

The feature that won me over: falling from a span into its logs

In SigNoz, the span details panel has a Logs tab. I clicked it with the slow span selected.

The money shot. There they were. Not "logs from around that time" — the logs of that exact span, joined by the trace_id the instrumentation had been stamping all along, with the two WARN lines glowing amber in the panel:

WARNING search_tool: upstream timeout, retrying (1/3)
WARNING search_tool: upstream timeout, retrying (2/3)
INFO    search_tool: succeeded on attempt 3

The same story in the Logs Explorer, filtered to trace_id = '359c33c37996f84f6cf2fc02ffd5ab03': the full narrative of one agent run, from "received question" through the auto-captured httpx lines of the actual Gemini API calls, to the retry loop, to the answer — each LLM step even reporting its token usage (llm plan completed (14 in / 196 out tokens)).

And there was my bug, in three lines: my search tool had a silent retry loop — a 3 s timeout and an aggressive exponential backoff against a flaky upstream. It always recovered by attempt three, so no error ever surfaced. The agent "worked." The user just waited 21 seconds. Plot twist: the villain was never Gemini. The villain lived in search_tool.py, three retries deep, wearing an INFO-level trenchcoat. Only the combination — trace for where, logs for why, welded by a shared ID — told the whole story, in roughly two clicks and zero timestamp archaeology.

The fix was three constants (cap the retries, shrink the backoff, fall back to cache fast). Next runs:

run 1: arithmetic: 2 + 2 = 4 (cached)  (3.6s)
run 2: arithmetic: 2 + 2 = 4 (cached)  (2.6s)
run 3: arithmetic: 2 + 2 = 4 (cached)  (2.7s)

~21 s → ~2.6 s. But the fix isn't the point. The two clicks are the point.

Why this works: one database, not three

This isn't UI sugar — it falls out of an architectural decision, and it's worth understanding:

Context propagation. OpenTelemetry stamps every log emitted inside a span with trace_id/span_id. The link lives in the data itself.
One datastore. SigNoz keeps traces, logs, and metrics in the same ClickHouse database. "Logs for this span" is a single indexed query — not a federated join across a trace store, a log store, and a metrics store with three query languages and three clocks.
One query engine. The same query builder speaks to all three signals, so correlation is a first-class action rather than a mental join you perform across browser tabs.

	Stitched-together stack (Tempo + Loki + Prometheus)	SigNoz
Trace → logs	Copy timestamp, switch tool, guess the window	Click the span's Logs tab
Query languages	TraceQL + LogQL + PromQL	One query builder
Clock skew between signals	Your problem	Same store, same clock
Time to "why"	Minutes, if you're good	Seconds

I've done the copy-the-timestamp dance for years. I didn't register how much of it was pure tax until SigNoz deleted it — turns out I'd been paying rent to three different landlords (Tempo, Loki, Prometheus) who don't talk to each other and definitely don't split a security deposit.

And the rest of the product hangs off the same spine, which is what makes the exploration feel coherent rather than like five bolted-together tools: the Services page derived RED metrics (P99, error rate, ops/sec) for my agent purely from its spans; the Logs Explorer runs fast structured queries over the same store (severity and trace filters, plus one-click Create an Alert / Add to Dashboard straight from any query); and because dashboards and alerts are built on the identical query builder, the p95-latency alert for this agent is the exact query I already wrote, reused.

To prove that to myself, I finished by building an Agent Health dashboard — four panels, each just a query-builder expression: agent runs over time (count() on the root span), p95 agent latency (p95(duration_nano)), LLM output tokens (sum(gen_ai.usage.output_tokens) — the GenAI conventions again), and tool retry warnings (count() on WARN logs). Four different questions, three different signals, one query language.

Read the top-right and bottom-right panels together and the whole incident is visible at a glance: p95 latency pinned above 20 s exactly while the retry-warning counts spike, then both collapse after the fix. That's the correlation thesis of this post, drawn as two lines — and it's four panels, zero SQL, and a distinct lack of me googling "how do I join traces and logs by timestamp" at 1 a.m., possibly ever again.

One more thing: I let an AI agent read my telemetry

SigNoz shipped an official MCP server — a Model Context Protocol bridge that lets any MCP-compatible AI assistant (Claude Code, Cursor, Gemini CLI…) query your observability data with natural tool calls. Since this hackathon is literally called Agents of SigNoz, I couldn't resist closing the loop: an AI agent debugging my AI agent.

Setup was one Docker command plus a read-only service account (Settings → Service Accounts — the MCP server only needs to read):

docker run -d -p 8000:8000 \
  -e TRANSPORT_MODE=http \
  -e SIGNOZ_URL=http://host.docker.internal:8080 \
  -e SIGNOZ_API_KEY=<service-account-key> \
  signoz/signoz-mcp-server:latest

The gotcha that cost me half an hour: I created the service account through the API (POST /api/v1/service_accounts) and passed a role field. The call returned 201 Created and the role was silently dropped on the floor — so every single MCP call came back 403 with "only viewers/editors/admins", which reads like a key problem when it's actually a role problem. I spent a genuinely embarrassing amount of that half hour convinced my API key was cursed. It was not cursed. It was roleless, which, philosophically, might be worse. The fix isn't in the request body: open the account in Settings → Service Accounts → Roles, add signoz-viewer, save. Then it works. If you're 403ing with a key you're certain is correct, go look at the role in the UI.

How many tools does it expose? The docs currently say 33. The tools/list call against the signoz/signoz-mcp-server:latest image I pulled returned 41: signoz_list_services, signoz_search_traces, signoz_search_logs, signoz_query_metrics, even signoz_create_alert and signoz_create_dashboard. This thing ships fast enough that any count in a blog post, this one included, is a snapshot. Ask your own instance rather than trusting my number.

I connected it to Claude Code and watched it re-run my entire investigation without touching the UI:

"What services do I have?" → signoz_list_services: one service, warmup-agent, 22 calls, p99 21.7 s, zero errors. (Zero errors! The agent is "healthy," remember?)
"Show me the slowest agent runs" → signoz_search_traces with duration_nano > 15s: trace 359c33c3…, 20.89 s — the exact trace from the flamegraph above, returned with a clickable webUrl.
"Any warnings?" → signoz_search_logs with severity_text = 'WARN': the retry-loop warnings — and because OTel logging captures code attributes, each log came back with code.function.name: "search_tool" and code.line.number: 122.

Read that last one again: the assistant didn't just find the warning — the telemetry told it which function and which line of my source code produced it. An AI agent read my observability data and pointed at line 122 of agent.py. My own code, ratted out by my own telemetry, to my own AI assistant. There's a lesson in there about accountability that I've chosen not to examine too closely. That's the "agent-native observability" pitch, working end-to-end on my laptop, against a self-hosted instance, with a read-only key.

Why this matters double for agents

Here's the thesis I'm carrying into the hackathon: for AI agents, signal correlation isn't a nice-to-have — it's the whole game.

An agent run is a trace: plan step, tool calls, reflection, answer — a tree of spans. The GenAI attributes annotate each step with model, tokens, and soon cost. The correlated logs record what each tool actually did. Put together, a single trace turns a non-deterministic black box into a readable story of one decision. My 18.3-second retry loop is the tame version; the same two-click move catches the expensive stuff — the agent that calls the same tool four times, the prompt that ballooned to 30k tokens, the tool that quietly falls back to stale cache.

Traditional services fail loudly. Agents fail politely. You need the trace to notice, and the logs to understand.

Three things I now believe in the way you believe things after they've personally embarrassed you:

"No errors in the logs" is not the same claim as "nothing is wrong." It is the claim "nothing threw an exception," which is a much smaller and much less comforting claim.
If your span is named after fire, maybe check that one first.
Retries are like glitter: a little bit hides a lot of problems, and you will regret using them within the hour.

Reproduce this yourself

Everything above is one repo and about five minutes. You need Docker and Python 3.11+; an LLM key is optional, because the agent falls back to a local stub with realistic latency if it doesn't find one.

# 1. Self-host SigNoz
curl -fsSL https://signoz.io/foundry.sh | bash
git clone https://github.com/wiz-abhi/Signoz && cd Signoz
foundryctl cast -f warmup-agent/casting.yaml

Now open http://localhost:8080 and create your admin account before doing anything else — that's the OpAMP gotcha from earlier. Skip this and step 3 fails with connection resets.

# 2. Install the agent's dependencies
cd warmup-agent
python -m venv .venv
.venv/Scripts/pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http openai  # Linux/macOS: .venv/bin/pip

# 3. Run it — buggy first, then fixed
export GEMINI_API_KEY=...                    # optional; omit to use the stub
.venv/Scripts/python agent.py --runs 3       # the slow version, ~21s per answer
.venv/Scripts/python agent.py --fixed --runs 5  # after the fix, ~2.6s

Then open the Traces Explorer, filter durationNano >= 15s, click the fat tool search_tool span, and hit its Logs tab. That's the whole post, replayed on your machine in the same two clicks.

Two things that will bite you on Windows specifically: run the Foundry install script from Git Bash, not PowerShell (it's a bash script), and use .venv/Scripts/ where the snippets above say .venv/bin/. The --fixed flag only moves three constants in agent.py — max_retries, base_backoff, and timeout — so you can watch the flamegraph collapse in real time.

What I'm building for the main event

This warm-up is becoming my Track 1 foundation: a multi-step, tool-calling agent with observability designed in from line one — per-step token and cost tracking via the GenAI conventions, alerts on runaway loops and token spikes, and a dashboard where every panel drills down to the exact trace (and therefore the exact logs) behind it. And now that I've seen the MCP server work, the loop I really want to build is self-referential: an agent that watches its own SigNoz telemetry and adjusts its behavior when it sees itself misbehaving. Whether I can pull that off in a week is an open question. That's what the week is for.

If you're on the fence about the hackathon: foundryctl cast, point anything you're building at localhost:4318, open your slowest trace, and click the span's Logs tab. That two-second fall from where to why is the moment "three pillars of observability" stops being a conference slide and becomes a debugging reflex.

Everything here ran on my laptop: self-hosted SigNoz v0.132 via Foundry, OpenTelemetry Python SDK, Gemini (gemini-3.1-flash-lite) as the agent's brain, the SigNoz MCP server in Docker, and a ~140-line agent. Code, scripts, and all screenshots: github.com/wiz-abhi/Signoz. Written for the Agents of SigNoz warm-up challenge by @wemakedevs and SigNoz.

AI assistance disclosure. I used Claude as a coding assistant while building the agent and as an editor while structuring this post. Everything it describes is mine and actually happened: I ran the stack, hit the bugs, took every screenshot from my own SigNoz instance, and verified each claim against the live system or the linked docs before it went in. The trace IDs and numbers are real — you can reproduce them with the steps above. Claude also appears in the story, as the MCP client in the last section; that part was the fun of it.

Building Kavach: catching fraud rings, not just fraudsters — with Neo4j

Abhishek Gupta — Wed, 08 Jul 2026 10:03:55 +0000

▶ Live demo: https://kavach-woad-beta.vercel.app
💻 Code: https://github.com/wiz-abhi/Kavach
🎥 Demo video: https://youtu.be/EZZLWpmqrS0

TL;DR — Traditional fraud detection scores one account at a time and misses organized rings. I modeled accounts, devices, IPs, phones, and transactions as a graph in Neo4j AuraDB, detected rings as dense infrastructure-sharing clusters with native Cypher, and got 100% recall / 0 false positives against ground truth. Then I made the "why a graph database?" argument provable by running the same query in Cypher vs. SQL, live.

The problem that started it

Every fraud system I'd read about asks the same question: "Is **this* account suspicious?"* It looks at one account's transaction amount, its velocity, its location, and scores it in isolation.

That works for lone fraudsters. It completely misses the expensive kind of fraud: organized rings. A ring is a group of accounts that quietly share infrastructure — the same device fingerprint, the same IP, the same phone number — and cycle money between themselves to launder it or farm signup bonuses and loans. No single account looks that bad. The pattern between them is the crime.

And that pattern is exactly what a row-and-column database is worst at. "Which accounts are secretly connected, and how deep does the ring go?" becomes a nightmare of self-joins in SQL. But it's the single most natural thing in the world for a graph.

So I built Kavach (Sanskrit for shield): model accounts, devices, IPs, phones and transactions as a graph in Neo4j AuraDB, then detect the rings as dense, infrastructure-sharing clusters — and explain why each one was flagged, in plain English.

The core idea: shared infrastructure is an edge

The whole thing hinges on one modeling decision. In Kavach, a device isn't a column on an account — it's its own node. When two accounts use the same device, they're two hops apart in the graph:

(ACC-402)-[:USED_DEVICE]->(DEV-401)<-[:USED_DEVICE]-(ACC-406)

Suddenly "find accounts that share infrastructure" is a graph pattern, and "trace the whole ring" is a variable-length traversal. Detection becomes: find clusters of accounts linked through shared devices/IPs/phones that also transact densely with each other — because sharing a device with your spouse is innocent; sharing a device and moving money in a tight loop is not.

What I actually built

Three services around one AuraDB instance:

Generator — synthetic accounts + deliberately seeded fraud rings, with a ground_truth.json so I could measure whether detection actually works.
Backend (Express + neo4j-driver) — detection, a live WebSocket feed, and the features below.
Frontend (Next.js) — a dark "security ops" dashboard with a live force-directed graph, ring alerts, and the interactive tools.

The moment the detection number went from "meh" to "wow"

My first detection run caught 1 of 4 seeded rings. Not great.

The bug was subtle and worth sharing: I was scoring ring "density" as internal_transactions / ring_size and comparing it to a per-account average. That metric punishes bigger rings — a 9-account ring looked less dense than a 6-account one, even when both were equally coordinated.

The fix was to measure transaction edge density relative to the graph's random baseline: of all possible pairs inside the cluster, how many actually transact, versus how often any two random accounts in the network transact. That number is size-independent and brutally discriminating. The seeded rings came back at 63× to 144× the baseline; the detection threshold sits at 5×.

Result, verified against ground truth: 100% recall, 0 false positives. I wired that check into npm run validate so I can prove it live instead of just asserting it.

Making the "why graph?" case provable, not just claimed

Judges on a database track have heard "graph is better for this" a hundred times. I wanted to show it. So Kavach has a "Why Graph?" panel that runs the identical fraud query two ways on the same live data:

In Cypher, against Neo4j.
In SQL, against an in-memory SQLite mirror of the same graph, built on the fly.

It displays them side by side and confirms the result sets match. On a single hop they're comparable — but the honest, interesting part is the transitive query ("trace the entire ring"): in Cypher it's one variable-length pattern; in SQL it's a recursive CTE that gets heavier with every hop. Same answer, very different query. That's the whole argument for a graph database, made concrete.

Two features that turn a red flag into an investigation

Detecting a ring is step one. A real analyst needs to act.

Investigate uses Neo4j shortestPath to answer "how are these two accounts secretly connected?" In "shared-identity" mode it ignores transactions and surfaces the hidden chain — e.g. ACC-402 shares an IP with ACC-401, who shares a device with ACC-406 — a link between accounts that never once transacted directly.
Analyst Copilot lets you ask in plain English ("How much money moved inside the rings?") and answers with a real, read-only Cypher query it generates and runs. It uses an LLM when configured, but falls back to a deterministic rule-based translator so a live demo can never be broken by an external API being down — and it shows you the Cypher every time.

Lessons from shipping it

Never trust an unrun build. The scaffolding "looked" done, but the graph endpoint was passing a float to LIMIT (which Neo4j rejects) and a stats query had a Cartesian product reporting 545,292 flagged accounts. Both would have died on stage. Run everything, end to end, before you believe it.
Measure your own detection. Seeding ground truth and reporting recall/precision turned a vague "it finds rings" into "4/4, 0 false positives." That one number is the most convincing thing in the whole demo.
Design your demo to not depend on the internet. Every impressive-but-fragile feature got a reliable fallback.

Try it

Kavach runs entirely on the Neo4j AuraDB free tier — detection deliberately avoids the GDS plugin (unavailable on free) so anyone can spin up an instance and run it. Full setup, API reference, and a 3-minute demo script are in the README.

Built for HACKHAZARDS '26 by NAMESPACE, on the Neo4j AuraDB track.

Catch the ring, not just the account.

I gave my AI agent's memory a CI/CD pipeline

Abhishek Gupta — Sun, 05 Jul 2026 18:16:09 +0000

🧠 I gave my AI agent's memory a CI/CD pipeline

Building SOBER — tests, diff, bisect, and gated deploys for a knowledge graph — for the Cognee × WeMakeDevs "Where's My Context?" hackathon.

TL;DR — Your agent's memory is production infrastructure with no tests, no diff, no rollback. SOBER is a brain CLI + GitHub Action that wraps a Cognee knowledge graph in real CI/CD: forget-regression tests that prove a retracted secret stays gone, git bisect for a poisoned graph, and a nightly improve() that opens its own pull request behind a green eval gate. Built solo, AI-assisted, and — the fun part — reviewed by a fleet of agents that found 14 real bugs in my own code.

🍸 The 3 a.m. problem

Every on-call engineer knows the feeling the hackathon is named after: it's 3 a.m., something is broken, and nobody remembers how it got fixed last time.

We solved that for code decades ago — version control, tests, code review, rollback, canary deploys. But your AI agent's memory, which increasingly decides what your agent knows and does, ships with none of it. It mutates in place, silently. There is:

❌ no test to stop a retracted secret or a stale, dangerous fact from staying recallable
❌ no diff to see what a re-ingest or an improve() run changed in the graph
❌ no bisect to find which ingestion poisoned the brain
❌ no gate before memory "ships" to production

So I built SOBER — CI/CD for Agent Brains. Not memory for DevOps. DevOps for memory.

🎯 Picking the idea (the honest part)

I didn't land on SOBER first. My initial instinct was an incident-memory copilot — an on-call assistant that recalls past outages. It felt strong until I pressure-tested it against the audience:

WeMakeDevs is a DevOps community, so "3 a.m. incident memory" is the single most predictable thing to build for them — and to Cognee's own engineers it reads as their past "Company Brain" hackathon winner with the nouns swapped.

It would have drowned in look-alikes. The winning reframe came from a different question: not "what memory app should I build?" but "what does memory-the-category still lack?"

The answer: the entire operations layer. Every other entry would use Cognee to remember things. SOBER governs the remembering itself — and in doing so it leans hard on the one Cognee verb nobody demos: forget().

🏗️ Architecture: a brain is a family of datasets

The load-bearing design decision. Cognee's forget() is scoped to a whole dataset, but I needed to retract one batch of knowledge without disturbing the rest. So a logical brain is modeled as a family of physical datasets — one per ingestion batch (node_set):

flowchart TD
    K["📄 knowledge/*.md"] -->|brain build| ING["ingest_batch()"]
    ING --> C["brain__core"]
    ING --> R["brain__runbooks"]
    ING --> X["brain__retracted 🔴"]

    subgraph BRAIN["🧠 logical brain = union of the family"]
        C
        R
        X
    end

    BRAIN -->|"CHUNKS recall (local, no LLM)"| EV["🧪 memory CI evals"]
    BRAIN -->|"export + merge"| SNAP["📸 snapshot vN (JSON)"]
    SNAP -->|diff| DIFF["🔀 graph diff"]
    X -.->|"forget(node_set='retracted')"| GONE["💨 surgically removed"]

    EV --> GATE{"🟢/🔴 gate"}

recall() and export() span the whole family → knowledge is found no matter which batch holds it.
forget(node_set) drops exactly one member → surgical retraction and one-batch rollback.
Membership is tracked deterministically in snapshots/.family.json.

That one indirection is what makes retraction and bisect-revert precise.

🔒 Capability 1 — Forget-regression tests

A production launch code gets ingested, then retracted. The memory-CI suite proves it's gone — not just from the obvious query, but across paraphrase probes and as residue in the exported graph.

Here's the actual run on real Cognee 1.2.2 + Gemini:

$ brain test          # secret still present
🔴 FAIL — brain failed memory CI
   6/11 passed, 5 failed
   🔒 forbidden  "what is the launch code"                 LEAK
   🔒 forbidden  "emergency launch authorization creds"    LEAK
   🧬 structure  no_node_text_matches /BRAVO-DELTA-\d+/     residue

$ brain revert brain__retracted        # forget(node_set), memory_only

$ brain test          # after surgical retract
🟢 PASS — brain is SOBER
   11/11 passed, 0 failed

$ brain diff
🔴 16 nodes / 32 edges removed   ← the retracted subgraph, nothing else

A retracted fact that stays retracted, proven on every change. No other memory tool ships that guarantee. The very first thing I validated — before writing a single feature — was that cognee.forget() genuinely removes a fact from vector recall (recallable → []), not just hides it. That one green check is what made the rest worth building.

🪓 Capability 2 — `git bisect` for a poisoned brain

When an eval goes red, some ingestion batch did it. SOBER binary-searches the batch history and pins the culprit in O(log n) probes:

$ brain bisect --failing-eval no-cache-flush-advice
   probe 1: prefix_len=8  full-set sanity      red=True
   probe 2: prefix_len=5  bisectbrain__b05     red=False
   probe 3: prefix_len=7  bisectbrain__b07     red=True
   probe 4: prefix_len=6  bisectbrain__b06     red=True
   >>> CULPRIT: bisectbrain__b06   (4 probes, linear would be 8)

$ brain revert bisectbrain__b06     # surgical forget → 🟢 green

A subtle correctness point I got to appreciate: bisect is inherently a forbidden-eval concept. It finds the batch that introduced a leak/poison, which stays present in every larger prefix (monotonic). A missing must_know fact doesn't work that way — so SOBER restricts bisect to forbidden evals rather than silently returning a wrong answer.

🌙 Capability 3 — The brain that ships itself

cognee.improve() distills chat sessions into the graph — a silent mutation that can regress memory. SOBER only runs it behind a green gate:

sequenceDiagram
    participant N as 🌙 nightly job
    participant CI as memory CI (evals)
    participant IM as cognee.improve()
    participant PR as pull request

    N->>CI: run evals BEFORE
    alt already red
        CI-->>N: ❌ refuse — never distill into a broken brain
    else green
        N->>IM: improve (distill sessions)
        IM-->>N: graph mutated
        N->>CI: run evals AFTER
        alt regressed
            CI-->>N: 🔴 block — no PR opened, report rollback snapshot
        else still green
            N->>PR: 📬 open PR (graph diff + before/after scores)
            PR-->>N: human merges to "deploy" the smarter brain
        end
    end

before	after	outcome
🟢 green	🟢 green	accepted — exit 0
🟢 green	🔴 red	blocked — no PR, rollback snapshot reported (exit 1)
🔴 red	—	refused — never distills into a broken brain (exit 1)

That's the CD half of CI/CD for agent brains: memory that proposes its own upgrades, behind a green gate and a human approval.

🧩 How it maps to Cognee

Every memory verb is load-bearing — including the rarely-used ones:

SOBER capability	Cognee API
Build the brain from source	`cognee.add()` + `cognee.cognify()`
Query for tests (keyless — local embeddings)	`cognee.search(SearchType.CHUNKS, datasets=family)`
Snapshot / diff the graph	`cognee.export(format="json")` → `{nodes, edges}`
Forget-regression / retraction	`cognee.forget(dataset=…, memory_only=True)`
CI-gated self-improvement	`cognee.improve(dataset, session_ids)`

🔥 The hardest part wasn't code — it was 20 requests a day

Gemini's free tier on an unbilled key turned out to be capped at 20 requests per day. I burned through it proving the core loop live, then discovered a fresh key just gets its own tiny 20/day — whack-a-mole.

So I adapted: the core forbidden-knowledge → forget loop is proven live, and I validated the trickier bisect and improve-gate logic with deterministic offline harnesses (stubbed verdicts, zero API calls). A good reminder that the interesting engineering is often in working around the constraint, not through it — and that a keyless test you can run 1,000 times is worth more than a live one you can run 20.

🤖 Building — and reviewing — with a fleet of agents

I built this with Claude Code as a pair programmer, and leaned in hard. After pinning down the real Cognee API with a few validation gates, I wrote a frozen interface contract and fanned out seven agents in parallel — one per module (the cognee wrapper, snapshot/diff, the eval suite, bisect, the CLI, the corpus, the workflows) — plus an eighth to integrate and import-check the assembly. It caught its own wiring bugs.

Then the part I'm proudest of: I turned the agents on my own code. A six-reviewer adversarial review — each finding re-verified by a skeptic that re-read the actual code before it counted — surfaced 14 confirmed bugs, zero false positives, deduping to 5 real root issues. The two nastiest:

- brain.improve(dataset="brain")          # targets the always-EMPTY base dataset
+ for member in list_family(dataset):      # spans brain__core, brain__runbooks, …
+     await cognee.improve(dataset=member) # the self-improve feature was a live no-op!

- glob("snapshots/brain__*")   # per-batch snapshot files that are NEVER created
+ read(".family.json")         # the registry that actually records the batches
                               # → `brain bisect` was dying every time

Both were in paths I'd only proven offline, so the stubs had masked them. I fixed all five, re-verified keyless, and pushed. Being able to adversarially review your own work — and have it find real bugs — is a genuine superpower.

(Per the hackathon's disclosure rules: AI assistance was used throughout; every result I call "proven" was executed against real Cognee + Gemini, not generated.)

🗺️ What's next

Cognee Cloud canary deploys — after a green brain ci, push() the brain to Cognee Cloud and serve() it to a slice of traffic, promoting or rolling back on live feedback.
In-process snapshot restore so a regressing improve() auto-rolls-back instead of only blocking.
More eval kinds — semantic contradiction detection, freshness/TTL checks, per-node-set coverage gates.

But the thesis is already standing: your agent's memory is production infrastructure, and now it can be tested, diffed, bisected, gated, and deployed like any other build artifact.

Your brain can't merge a regression anymore.

⭐ Repo: https://github.com/wiz-abhi/SOBER · Built on Cognee for The Hangover Part AI