One of the example projects listed for this hackathon was basically "self healing infra." Alert fires, agent restarts a container, done. I figured half the submissions were going to be some version of that, so I didn't want to build it. Also, honestly, it didn't feel like it would prove anything. An agent reacting to a threshold isn't the same as an agent understanding what actually happened.
So instead of one service with one alert, I built four, chained them together, and then broke one of them on purpose in a way that would trick anyone (or anything) that wasn't actually looking at the trace properly.
The chain
Order calls Payment, Payment calls Inventory, Inventory calls Notification. Four FastAPI services, one request fans out through all of them over httpx.
Then I did something mean to myself. I added a chaos flag to Payment that adds a random 2 to 5 second sleep to about 30% of requests, to simulate a flaky downstream dependency.
Here's the part that made this actually worth building. When Payment is slow, Order looks slow too, because Order is just sitting there waiting on it. If you only look at "which service had the longest request," you'll blame whichever one happens to be closest to the client, which in my case would be Order, and that's the wrong answer.
Getting one trace instead of four
The non negotiable part of this build was that a single request had to show up in SigNoz as one connected trace across all four services, not four separate ones that just happened to be related.
I self hosted SigNoz through Foundry, same as my pre event blog, then added opentelemetry-instrumentation-fastapi and opentelemetry-instrumentation-httpx to each service and pointed the OTLP exporter at my existing deployment. The thing nobody warns you about is that if you create your httpx.AsyncClient before the instrumentation is wired up, or if the async context gets dropped anywhere in the chain, you just quietly get four disconnected traces instead of one. No error, nothing obviously wrong. You only notice it if you go into SigNoz and check whether the trace IDs actually match across services.
My root cause agent was confidently wrong once
Once traces were flowing and I had an alert and a dashboard panel showing the cascade shape, I built the actual agent. It pulls a trace through the SigNoz MCP server, hands the span data to Gemini 2.5 Flash, and asks it to name the root cause.
First version just fed Gemini each span's total duration. Worked fine when Payment's delay was obviously the biggest number in the trace. Then on one run where the trace was only a little slow (normal network jitter, chaos hadn't actually triggered), Gemini confidently blamed Notification. Not because it was reading the data wrong. Notification's own span really was the longest number on the page. It just didn't know that "longest span" and "root cause" aren't the same thing.
That's the actual lesson I got out of this, and it's not something either the SigNoz docs or the OpenTelemetry docs tell you directly. A span's total duration includes time spent waiting on whatever it called downstream. So a service can look slow purely because something below it is slow. That's not evidence it's guilty. If anything it's evidence it's innocent.
Self time fixed it
The number that actually matters is self time, how long a span took doing its own work, not counting time spent waiting on its children. I computed it straight from the trace data:
self_time_ms = span.duration_ms - sum(child.duration_ms for child in span.children)
Sorted the spans by self time before sending them to Gemini, and rewrote the prompt to tell it explicitly that self time is the signal to trust, not raw duration. I also bumped my trace search filter from 1 second to 2 seconds, since anything under that was just normal noise, not my actual injected chaos.
After both changes the diagnosis got reliable. Payment's own span consistently had the highest self time, and the agent named it correctly every time I reran the load test, not just on the runs where the chaos delay happened to be the single biggest number around.
Restarting it and then actually proving it worked
Diagnosing the problem isn't the interesting part if the agent doesn't do anything about it. Once Payment is identified, the agent hits an admin endpoint on Payment to disable the chaos flag, standing in for restarting the actual failing dependency. Then it sends 10 fresh requests through the whole chain and times them itself.
Before the fix, requests were taking 2000 to 5000ms because of the cascade. After, fresh requests averaged 46.12ms with a max of 53.91ms. Back to normal.
One thing worth admitting. I also queried SigNoz for Payment's P95 as a second check, and that number came back higher than my fresh measurements, because its default window still had all the earlier chaos affected traffic mixed in with the clean requests. My own timed requests were the real proof. SigNoz's number was just supporting evidence, diluted by old data. Figuring out which of my two "proof" numbers to actually trust mattered more than any single line of code in this whole project.
What I actually think this proves
Anyone can make an alert trigger a restart and record a demo that looks impressive. What I wanted to show is that the agent's diagnosis is actually grounded in the trace data instead of getting lucky on which number happened to be the biggest. Hitting that self time bug and having to fix it is honestly the part of this project I'm most glad happened, because it's the difference between an agent that looks right and one that's actually reasoning correctly.
Stack
FastAPI, httpx, OpenTelemetry Python SDK, SigNoz self hosted through Foundry, Docker Compose, the SigNoz MCP server, Gemini 2.5 Flash on the free tier, and Docker for the remediation step.
AI tool disclosure: GitHub Copilot for in editor code, Claude for architecture and debugging help while building this. Gemini 2.5 Flash is the model used inside the actual project for root cause reasoning, not for writing the code, it's a real part of the running system.
Repo, with casting.yaml and casting.yaml.lock included: https://github.com/shiv-codez/cascade-detective
(I used Claude to help with debugging guidance, structuring this post, and editing for clarity, as disclosed per the hackathon rules. All setup, exploration, and technical decisions described here were done by me.)
Top comments (0)