I built an AI agent that fixes incidents and proves it — using SigNoz

How my team built a self-healing SRE agent for the Agents of SigNoz hackathon: it detects a problem in SigNoz, fixes the app, and then uses SigNoz again to prove recovery.

---

Every observability tool — including SigNoz's own new MCP server — is great at one thing: telling you what broke. "This endpoint is slow, here are the traces." But then a human still has to read the data, guess the cause, apply a fix, and check whether it actually worked.

For the Agents of SigNoz hackathon (Team TraceBandits — Jay Bamroliya & Kaushal Karkar), we asked a different question: what if the agent did that whole loop? Not just explain the incident, but fix it — and then prove the fix worked, using observability.

That last part is the whole idea. The result: an endpoint that was taking 4.2 seconds drops to tens of milliseconds, detected → fixed → verified, with one click and no human in the loop.

The agent's dashboard right after a heal: it detected the slow /about endpoint, disabled the fault, and re-tested it — 4.21 s → 36 ms, with its own incident report.

What it does

The agent runs a closed loop:

1. Detect — reads service metrics from SigNoz and spots the degraded service.
2. Investigate — pulls per-operation latency (including internal DB spans) to find where the time actually goes.
3. Fix — applies a scoped remediation to the running app.
4. Verify — re-tests the endpoint and confirms recovery, then writes its own incident report.

Most "AI + observability" demos stop at step 2. Steps 3 and 4 — acting and then verifying via observability — are what make this different, and they're exactly what an on-call engineer actually wants.

One click: I inject a latency fault, the app goes DEGRADED, and the agent starts investigating SigNoz live.

The architecture

The whole system: an OTel-instrumented app streams to SigNoz; the agent reads SigNoz through tools and acts back on the app.

• Agent: Spring Boot (Java). Most agents are Python; doing it in Java kept it close to the app we were healing and easy to reason about.
• Brain: Llama-3.3-70B on Groq (free tier, OpenAI-compatible function calling). The LLM decides which tools to call and in what order.
• Observability: SigNoz (Cloud for the demo; the repo ships casting.yaml so you can self-host it with Foundry).
• Demo target: an OpenTelemetry-instrumented Spring Boot app with a runtime fault-injection control plane, so we can stage a real, controllable incident.

How the agent uses SigNoz

This is the heart of the project, so here are the actual tools the LLM can call — each is a call to the SigNoz Query API (authenticated with a service-account key):

• get_services → POST /api/v1/services — p99 latency, error rate, throughput per service. Finds the degraded service.
• get_top_operations → POST /api/v1/service/top_operations — p50/p95/p99 per operation, including internal spans (DB calls, repository methods). This is how the agent pinpoints where the latency lives, not just that a request is slow.
• apply_fix → calls the app's control plane to disable the fault.
• verify_recovery → re-tests the endpoint directly and reports the new latency.

Notice SigNoz is used twice in the loop: once to detect and diagnose, and — after the fix — again to confirm recovery. That verification-via-observability step is the part I'm proudest of.

SigNoz showing the /about endpoint's latency — this is what the agent reads to find the bottleneck.

Here's a real, verbatim conclusion the agent wrote during a run:

"Incident summary: the 'smartcontactmanager' service had an endpoint '/about' with abnormally high latency (p99 ≈ 4.2 s). The fix applied was to disable the injected latency fault. After the fix, the endpoint was re-tested and its average response time was 36 ms, confirming recovery."

What broke while building it (the useful part)

1. Spring bean-name clashes. I registered RestClient beans named groqClient, demoAppClient — which collided with the @Component classes of the same name. Spring refused to start: "a bean with that name has already been defined." Fix: rename the @Bean methods (groqRestClient, demoRestClient) and match the constructor parameter names, since with multiple RestClient beans Spring disambiguates by parameter name.

2. Llama stringifies numbers. My first tool schema had an integer minutes parameter. Groq's validation kept rejecting the model's output: "expected integer, but got string" — Llama emitted {"minutes": "60"}. Fix: drop flaky numeric params; the tools take only strings and the time window is a server-side default. Simpler schema, zero validation failures.

3. Ingestion delay vs. real-time proof. SigNoz Cloud has a short ingestion lag, so right after a fix the aggregated p99 hasn't updated yet. For deterministic proof, verify_recovery re-tests the endpoint directly and reports the measured latency — instant and honest — while SigNoz's dashboards show the same recovery a little later.

What I learned

• Observability isn't just for humans anymore. The moment you expose telemetry through a clean API (or an MCP server), an agent can reason over it — and act on it.
• The verify step is underrated. Any agent can propose a fix. An agent that checks its own work against real signals is one you might actually trust.
• Free tools go far. Groq's free Llama + SigNoz + OpenTelemetry — the whole stack cost nothing and runs on a 6 GB laptop.

Try it

The code, the casting.yaml for a reproducible SigNoz deployment, a pitch deck, and a full demo video are all in the repo. Clone it, drop your SigNoz + Groq keys into .env, run the app and the agent, open the dashboard, and click Self-Heal.

SigNoz's MCP server lets an AI see your system. This agent lets it heal your system — and prove it did.

— Team TraceBandits · Agents of SigNoz (WeMakeDevs × SigNoz)