DEV Community: Shivam Singh

I Built a 4 Service Cascade Failure Just to See If an AI Agent Could Actually Diagnose It

Shivam Singh — Sat, 25 Jul 2026 20:22:16 +0000

One of the example projects listed for this hackathon was basically "self healing infra." Alert fires, agent restarts a container, done. I figured half the submissions were going to be some version of that, so I didn't want to build it. Also, honestly, it didn't feel like it would prove anything. An agent reacting to a threshold isn't the same as an agent understanding what actually happened.

So instead of one service with one alert, I built four, chained them together, and then broke one of them on purpose in a way that would trick anyone (or anything) that wasn't actually looking at the trace properly.

The chain

Order calls Payment, Payment calls Inventory, Inventory calls Notification. Four FastAPI services, one request fans out through all of them over httpx.

Then I did something mean to myself. I added a chaos flag to Payment that adds a random 2 to 5 second sleep to about 30% of requests, to simulate a flaky downstream dependency.

Here's the part that made this actually worth building. When Payment is slow, Order looks slow too, because Order is just sitting there waiting on it. If you only look at "which service had the longest request," you'll blame whichever one happens to be closest to the client, which in my case would be Order, and that's the wrong answer.

Getting one trace instead of four

The non negotiable part of this build was that a single request had to show up in SigNoz as one connected trace across all four services, not four separate ones that just happened to be related.

I self hosted SigNoz through Foundry, same as my pre event blog, then added opentelemetry-instrumentation-fastapi and opentelemetry-instrumentation-httpx to each service and pointed the OTLP exporter at my existing deployment. The thing nobody warns you about is that if you create your httpx.AsyncClient before the instrumentation is wired up, or if the async context gets dropped anywhere in the chain, you just quietly get four disconnected traces instead of one. No error, nothing obviously wrong. You only notice it if you go into SigNoz and check whether the trace IDs actually match across services.

My root cause agent was confidently wrong once

Once traces were flowing and I had an alert and a dashboard panel showing the cascade shape, I built the actual agent. It pulls a trace through the SigNoz MCP server, hands the span data to Gemini 2.5 Flash, and asks it to name the root cause.

First version just fed Gemini each span's total duration. Worked fine when Payment's delay was obviously the biggest number in the trace. Then on one run where the trace was only a little slow (normal network jitter, chaos hadn't actually triggered), Gemini confidently blamed Notification. Not because it was reading the data wrong. Notification's own span really was the longest number on the page. It just didn't know that "longest span" and "root cause" aren't the same thing.

That's the actual lesson I got out of this, and it's not something either the SigNoz docs or the OpenTelemetry docs tell you directly. A span's total duration includes time spent waiting on whatever it called downstream. So a service can look slow purely because something below it is slow. That's not evidence it's guilty. If anything it's evidence it's innocent.

Self time fixed it

The number that actually matters is self time, how long a span took doing its own work, not counting time spent waiting on its children. I computed it straight from the trace data:

self_time_ms = span.duration_ms - sum(child.duration_ms for child in span.children)

Sorted the spans by self time before sending them to Gemini, and rewrote the prompt to tell it explicitly that self time is the signal to trust, not raw duration. I also bumped my trace search filter from 1 second to 2 seconds, since anything under that was just normal noise, not my actual injected chaos.

After both changes the diagnosis got reliable. Payment's own span consistently had the highest self time, and the agent named it correctly every time I reran the load test, not just on the runs where the chaos delay happened to be the single biggest number around.

Restarting it and then actually proving it worked

Diagnosing the problem isn't the interesting part if the agent doesn't do anything about it. Once Payment is identified, the agent hits an admin endpoint on Payment to disable the chaos flag, standing in for restarting the actual failing dependency. Then it sends 10 fresh requests through the whole chain and times them itself.

Before the fix, requests were taking 2000 to 5000ms because of the cascade. After, fresh requests averaged 46.12ms with a max of 53.91ms. Back to normal.

One thing worth admitting. I also queried SigNoz for Payment's P95 as a second check, and that number came back higher than my fresh measurements, because its default window still had all the earlier chaos affected traffic mixed in with the clean requests. My own timed requests were the real proof. SigNoz's number was just supporting evidence, diluted by old data. Figuring out which of my two "proof" numbers to actually trust mattered more than any single line of code in this whole project.

What I actually think this proves

Anyone can make an alert trigger a restart and record a demo that looks impressive. What I wanted to show is that the agent's diagnosis is actually grounded in the trace data instead of getting lucky on which number happened to be the biggest. Hitting that self time bug and having to fix it is honestly the part of this project I'm most glad happened, because it's the difference between an agent that looks right and one that's actually reasoning correctly.

Stack

FastAPI, httpx, OpenTelemetry Python SDK, SigNoz self hosted through Foundry, Docker Compose, the SigNoz MCP server, Gemini 2.5 Flash on the free tier, and Docker for the remediation step.

AI tool disclosure: GitHub Copilot for in editor code, Claude for architecture and debugging help while building this. Gemini 2.5 Flash is the model used inside the actual project for root cause reasoning, not for writing the code, it's a real part of the running system.

Repo, with casting.yaml and casting.yaml.lock included: https://github.com/shiv-codez/cascade-detective

(I used Claude to help with debugging guidance, structuring this post, and editing for clarity, as disclosed per the hackathon rules. All setup, exploration, and technical decisions described here were done by me.)

What Broke When I Self-Hosted SigNoz on Windows

Shivam Singh — Sun, 19 Jul 2026 17:59:18 +0000

I just finished my first year of CS. I know basic Python and C, and right now I'm doing DSA in Java. Before this weekend I had never touched Docker, WSL, or anything related to observability. When I saw the pre-event blog challenge for the "Agents of SigNoz" hackathon, I thought it would be a chill way to warm up before the real thing started. It was not chill. That's basically the whole post.

Docker Desktop was the wrong move and I almost didn't catch it

I'm on Windows 11, so my first instinct was the obvious beginner move. Install Docker Desktop, click through it, done. Except SigNoz's own docs specifically say not to do this. There's a known bug where ClickHouse Keeper, one of SigNoz's core pieces, crashes when it runs under Docker Desktop's virtualization on Windows. Their fix is to skip Docker Desktop for actually running containers, and instead install Docker Engine natively inside WSL2.

I didn't even know what WSL2 was before this. It's basically a real Linux environment that runs alongside Windows. So I had to turn on virtualization, install Ubuntu through WSL, and then run Docker's install script from inside that Ubuntu terminal instead of using the normal Windows installer. The part that actually confused me was that Docker Desktop auto connects itself to WSL by default. So even after I installed the native engine correctly, Docker Desktop was quietly sitting there ready to override it. I had to go into Docker Desktop's settings and manually turn off "WSL Integration" for my Ubuntu distro to keep the two separate.

What I thought would take five minutes ended up eating close to an hour, mostly because I didn't know any of this going in. If you're on Windows and about to try this, just go straight for native Docker Engine in WSL. Don't let Docker Desktop anywhere near your containers.

Actually deploying SigNoz was the easy part

Once Docker was finally behaving, deploying SigNoz itself was almost anticlimactic. SigNoz uses something called Foundry as its official installer. You write a small casting.yaml file describing what you want (mine was just flavor: compose, mode: docker), run foundryctl cast -f casting.yaml, and it does the rest. Pulls images for ClickHouse, the SigNoz backend, the OTel collector, Postgres, and spins up around a dozen containers. It printed a clean list of everything that started and I didn't have to debug anything here.

One thing I liked as a beginner: the whole setup lives in that one casting.yaml file plus a casting.yaml.lock it generates automatically. That's what the hackathon judges will use to reproduce my exact deployment. I don't usually think about reproducibility so it was a good habit to see in action.

I skipped their demo app on purpose

SigNoz has an official demo app you can point at your instance and get realistic looking dashboards instantly. I decided not to use it. Running someone else's pre built app and screenshotting the output felt like it would teach me basically nothing. I wanted something that was actually mine to poke at. So I wrote a tiny Flask app instead. Three routes, /, /orders, /checkout. I added fake latency using time.sleep() so requests wouldn't all look identical, and made /checkout fail on purpose about 20% of the time, to simulate something like a flaky payment gateway.

Getting OpenTelemetry hooked up took way less code than I expected. I ran pip install opentelemetry-distro opentelemetry-exporter-otlp, then opentelemetry-bootstrap -a install, which scans your installed packages and auto installs the right instrumentation for whatever you're using. Then instead of running my app normally, I ran it like this:

opentelemetry-instrument python app.py

That's it. I didn't add a single line of tracing code to my own routes. This was honestly the biggest "wait, that's all?" moment of the whole day. I expected instrumentation to mean manually wrapping functions or adding decorators everywhere. Instead the tracing just wraps around your app from the outside.

Seeing my own data show up was genuinely satisfying

Once I hit my routes a bunch of times with curl, the SigNoz home page confirmed logs, traces, and metrics were all live. The Services panel showed real numbers pulled straight from my own traffic. P99 latency error rate, requests per second.

The part I spent the most time on was the trace detail view. I filtered specifically for one of my /checkout 500 errors and clicked into it. Full flame graph, a waterfall timeline, and a complete attributes panel with everything down to http.status_code: 500 and which client hit it. It correctly picked up curl/8.18.0, which is kind of funny because that's literally just me testing it from my own terminal. Since my app only does one thing per request, the waterfall was just one flat bar. But it made obvious how this would look on a real app with multiple services. You'd see nested spans, and you could tell exactly which downstream call, a database, a third party API, actually caused the slowdown or the failure. Reading about tracing never made that click for me. Seeing an actual flame graph did.

Dashboard and alert, and one annoying gotcha

I built a small dashboard with two panels. P99 latency grouped by route, and error count over time. The latency panel shows something real and visible. / responds in under a millisecond, while /checkout spikes to 300 to 400ms, which lines up exactly with the artificial delay I put in the code. The error panel isolated my one /checkout failure down to the exact second it happened, which felt oddly precise for something I built in an afternoon.

Setting up an alert (fire if errors go above 2 within a 5 minute window) is where I hit a small annoying wall. SigNoz won't let you save an alert rule at all unless it has a notification channel attached, even if you have zero intention of sending a notification anywhere. I ended up creating a fake webhook channel pointing at a placeholder URL just so it would let me save the rule. Small thing, but it cost me a few confused minutes, so noting it here in case someone else hits the same wall.

I also poked at the MCP server without a full AI client hooked up

SigNoz ships an MCP server, and it seems to be the actual headline feature of this whole hackathon. The idea is that an AI coding assistant can query your traces, dashboards, and alerts directly through natural language instead of you clicking through the UI. I enabled it through Foundry, just adding an mcp block to the same casting config, confirmed it was actually alive by hitting its health check endpoint, and set up a scoped service account with an API key for it. I didn't get as far as connecting it to a live AI client this time. That's a whole separate setup I ran out of time for. But getting the server running and authenticated end to end still taught me more about how it fits into a workflow than just reading the docs page would have.

Things I'd tell myself from yesterday

If you're on Windows, skip Docker Desktop entirely for this. Go straight to native Docker Engine inside WSL. Save yourself the hour.
Auto instrumentation really is close to free. I expected to write tracing code and I wrote zero.
Writing my own toy app instead of using SigNoz's demo cost me maybe 45 extra minutes, and it was worth every one of them. Describing your own bug just hits different than describing someone else's dashboard.
I had to actually look up what "P99 latency" means properly. It's the slowest 1% of requests, not "99% of requests." That distinction actually matters when you're staring at a dashboard trying to figure out if something's wrong.

Conclusion

I went into this weekend not even knowing what Docker Desktop was, and came out having self hosted a full observability stack, broken it once, fixed it, and built my own dashboards on top of data from an app I wrote myself. Most of what I actually learned came from things not working the first time, not from anything I read beforehand. If you're a beginner thinking about trying this challenge, budget more time for environment setup than feels reasonable, and don't take the shortcut of using the official demo app. The friction is genuinely where the learning is.