Well I wanted to reimagine the classic 2 AM page that every developer has gone through at least once, a production downtime that would rattle engineers out of their bed, most of them half-asleep, forced to stare at a dashboard, wondering what went wrong.Once they realised it wasn't a false alarm, they would run through a standard incident playbook , digging through logs, checking if an unannounced deployment went alive and than would systematically attack the outage. Today? It looks more a engineer still clueless about the failure, desperately tossing a broken production environment at an AI agent with a silent plea: "please help me fix this, make no mistakes.", thanks to LLM's :) .Well since we are officially in the era of handing zero context chaos to algorithms, I wanted to see if the reality lived up to the hype, minus the actual 2 AM wakeup call with an agent doing these analysis as soon as the notification hits. So, I broke my own payment processor on purpose, and watched three services fall over like dominoes. Well than as we have the routine I asked the agent that I had created, to tell me why. Before I could even finish opening the logs, the agent had already provided the entire series of actions that would have taken place. It pin-pointed the exact service service, provided the exact trace, and did it faster than I could have mapped it by hand.Let us walk through how I built it, broke it, and what the agent caught.
What I Built to Simulate This (and Why the Setup Matters)
Well since I can't purposefully break a production application to test out my agent skills (i know the title say's otherwise, but it is what it is :) ) I went on to create three services that talk to each other the way production actually does. These three services are as follows :
api-gateway* :8000) a public front door, a canary router: 80% of traffic to a stable orders-service v1, 20% gambled on an untested v2 - a selective release , not a full roll out.
orders-service* :8001 v1 / :8002 v2) that writes an order to SQLite, then synchronously calls payments to charge the card. If payments is slow, everything behind it waits.
payments-service* :8003) a fake third-party payment processor, with a secret control endpoint I built to introduce certain surprises.
All three are wired up with OpenTelemetry, the open standard for emitting traces (a timed, nested record of one request as it bounces between services), metrics, and logs wrapped each service in opentelemetry-instrument, zero code changes, shipped to a self-hosted SigNoz instance.
Standing SigNoz up was a simple affair, not the usual afternoon lost to broken Docker configs and missing dependencies.. Not this time. Within twenty minutes, I had a clean, open source telemetry pipeline ready.
curl -fsSL https://signoz.io/foundry.sh | bash
foundryctl cast -f casting.yaml # deployment: {flavor: compose, mode: docker}
This helped me setup the UI on localhost:8080, telemetry ingestion on 4317/4318, everything in under five minutes.
Then to simulate traffic that normally comes in during load, I used k6 ramping 1 to 15 virtual users over 11 minutes to mimic a real morning traffic curve:
- 5,814 requests, 245 failed (4.21%)
- http_req_duration: avg 418ms, median 103ms, p90 651ms
- p95: 3.26 seconds
That p95 means real users were waiting 3+ seconds, which where I needed to dig. It means a subset of these requests were having a bad time which in real world scenario would impact users.
The crime (enter the sabotage)
After running the load tests, at the third minute, I purposefully degraded the payments control endpoint by introducing a 2 to 4.5 second delay per request and dropping 40% of calls with an HTTP 503.
This was done to simulate what happens in real world, a third party provider or another team in the same organization deploys an upgrade to one of their services. It worked fine on their test cases (works on their machine :) ) but that service in reality starts , it consecutively triggers a massive slowdown across our main application. The system isn't dead, but it's suddenly limping and unreliable in a way that standard uptime checks completely miss until the blast radius has already spread.
Now as we have structured our architecture, api-gateway → orders-service → payments-service are all a synchronous call chain, the pain shot straight back up.The error rate for all three services and p99 latency spiked in the exact same 5 second buckets which tells that the three services failing in lockstep isn't three problems, it's one problem wearing three costumes.
After going through the dashboard ,I found a trace 0050c21ff1f619a14f9bbbb7f7a5a182, 11 spans, 8 errored. A span is one timed unit of work inside the request. :
That leaf span(the last span) tells a lot, payments POST /charge actually isn't dependent on any other service, it's the last api that this entire journey calls and it just sits for 2.73 seconds and fails. Each span's duration is within 40ms of its parent's which is actually the proof that this is a blocking cascade, not three independent slowdowns, each layer just waits on the one below it, then re-throws the failure on the way up (503 → 502 → 502).This realistically takes 20-30 minutes to figure out under normal conditions, but 2am it would have escalated fast..Now this is where I was always wanted an automated buddy to take my position, check these logs and bring up the analysis for the same till I would be up , gaining my senses. If this happened, it would ideally help a lot of developers.
So could an AI agent have caught the same thing?
Before going to this, SigNoz's cloud product ships an AI teammate called Noz that investigates incidents in plain English ,but since i had hosted Signoz locally, it wasn't available since its a cloud only feature provided by them. What was available was a plethora of tools consisting 40 tools spanning traces, logs, metrics, alerts, and dashboards exposed by Signoz via MCP. (MCP is just a standard way to hand an AI agent a toolbox it can call into.).Instead of bloating the context window of my agent (by giving it the code files and pointing to logs) and than saying it "please fix this, make no mistakes"I wired in an ordinary AI agent with exactly one instruction that it needs to use these tools in order to come to arrive to a conclusion. This is how it went:
"My api-gateway service is having latency and error issues in the last 15 minutes. Investigate by check traces, errors, and top operations for api-gateway, orders-service, and payments-service and tell me what's actually wrong.Use appropriate tools to come to this conclusion"
Eight MCP tool calls across two turns later, it came back with, close to verbatim:
"Root cause: payments-service is the failure point, everything else is a symptom. POST /charge in payments-service is failing with HTTP 503, taking 2.6 - 4.5s to fail (vs. its normal ~90ms p50)... All three services spiked simultaneously, in the same 5 second buckets, which rules out three independent problem, it's one upstream failure cascading through the call chain."
Well this was something that I had also arrived minus the 20 minutes. Further to narrow down the exact root cause, I asked : "Is this a cascading failure from payments-service? Show me the specific trace." It pulled the exact trace ID above, unprompted, walked the span tree hop by hop, and landed on "each hop is just blocking on the one below it, not doing independent slow work of its own", exactly what I'd concluded twenty minutes earlier.
I just want to reiterate : we had given this agent the same access levels that a human would have,no sandboxing, no special context just asked it to use MCP tools provided by Signoz.While answering the query, eight calls total across two turns, every one of them a SigNoz tool (signoz_list_services, signoz_aggregate_traces, signoz_get_service_top_operations, signoz_search_traces, signoz_get_trace_details), zero file reads.Also noting the fact that it always had the option of choosing in the two-second shortcut sitting right next to it and didn't take it; it got there by querying telemetry instead of opening main.py and spotting the exception I'd planted like a landmine (which ideally is seen when model takes in multiple turns to answer a query it goes on and would directly read the code instead of relying on the telemetrics).
Well the working was a sight to see ,it took responsibility and cross referenced operations, error rates, and trace timing across three services in about the time it took me to type the follow-up, because traces, metrics, and logs already lived in one queryable, OpenTelemetry native store. Well ideally it's always tempting to credit this to the model's intelligence, but that's not quite the entire story over here, a smarter model fed bad or incomplete telemetry would reason just as poorly as a weaker one. The quality of the underlying data mattered more than the model itself.
In the same session,the agent also caught a second bug I had planted , 5% of GET /orders/{order_id}/receipt calls on the v2 canary were quietly raising an unhandled RuntimeError. A simple health check would never catch it, 95% of requests still succeed, dashboard stays green. A single Query Builder filter service.name = 'orders-service' AND name = 'GET /orders/{order_id}/receipt' AND has_error = true) surfaced every occurrence, full stack trace attached, zero try/except written by me.
Everything doesn't go smoothly (that's reality of anything life or onboarding a new stack)
- Well after reading the documentation, writing up the code for the agent and wiring in all tools, my first MCP call came back with a flat 403. Reason : a service account with no role assigned meaning just i had to assign signoz-admin role, restarted the entire session since the mcp tools only load when an agent starts a session and voila it was fixed . The happiness was similar to when a toddler get's it's favourite toy to play with
So, What's the Takeaway?
AI isn't going to replace experience that engineers carry. But it can read a trace waterfall with the same reasoning as a human would , but faster than we can.
The real plot twist here isn’t that the LLM is has some kind of reasoning superior to humans (untill we have AGI or ASI) it's reasoning is ideally derived from human's intellect ; it’s that AI is only as useful as the telemetry foundation you feed it. The "AI teammate" pitch stopped being marketing fluff the second a generic agent, acting on raw data without any hints, handed me a flawless incident report before I could even finish opening the logs.
So why did it actually work ? Three things :
One store, not four: The agent answered "what's wrong across three services" in one pass because traces, metrics, and logs live in the same backend. There was no tool-hopping or manual timestamp alignment just to figure out the blast radius.
The AI tooling isn't locked behind a cloud tier: While Noz (SigNoz's builtin AI teammate) is cloud-only, the open-source Model Context Protocol (MCP) server that did the heavy lifting here is completely free. It runs locally and talks directly to a self-hosted instance without requiring an enterprise contract.
Getting instrumented cost me nothing: opentelemetry instrument wrapped FastAPI and httpx out of the box with zero code changes. The classic friction of "telemetry is too hard to set up" simply wasn't there.
I haven't run this same chaos test against Datadog or Grafana, so I won't claim SigNoz uniquely wins the observability crown. But I will say this: the debate over whether an agent can handle real, zero-context production failures is officially over. It beat me to the punch on my own crime scene.
Top comments (0)