I Built Four AI Agents That Fix Bugs by Themselves, Then Made Them Watch Themselves Work

Nagarjuna Bolla — Sun, 26 Jul 2026 18:32:32 +0000

We've gotten good at observing software. We're not yet good at observing the agents that increasingly write and operate that software.

When a service throws a 500, you open a trace and see exactly which span failed. When an AI agent does something wrong, you usually get… a log line saying it failed. Maybe a stack trace. Rarely the reasoning, the tool calls it made, what it cost, or — most importantly — whether the failure was the agent's fault at all.

So for the Agents of SigNoz hackathon I built AgentOps: four autonomous agents that detect, diagnose, fix, and verify a real bug in a real application with no human in the loop. The self-healing part is the demo. The actual project is that every thought those agents have is a span in SigNoz.

The setup

I deliberately did not scaffold my own toy app to break. Fixing a bug you planted in code you wrote is not convincing.

Instead I vendored BuggyBoard, an MIT-licensed bug tracker by Andrew Knight — Express, TypeScript, SQLite. Then I seeded exactly one defect into its not-found guard:

const bug = getBug(id);

if (!bug) {
if (!bug.id) { res.status(404).json({ error: "not_found", message: "Bug not found." });

Now GET /api/bugs/999999 dereferences null, throws a TypeError, and Express returns 500 where it should return 404. A boring, extremely realistic bug.

There's a nice symmetry to it: the agents fix a bug in a bug tracker.

The loop

Monitor  →  Diagnosis  →  Fix  →  Verify

Monitor polls SigNoz every four seconds asking "any error spans from demo-api since my last check?" It has no privileged backchannel to the application. It finds out the same way a human on-call engineer would — by querying the observability backend. It makes zero LLM calls; the threshold is deterministic. Its agenthood is the autonomous loop, not cleverness.

Diagnosis pulls the ERROR logs and the full trace from SigNoz, hands only that evidence to an LLM, and demands strict JSON back: file, line, cause. It answers src/index.ts:87. It never reads the source code, and it certainly never reads the answer key.

Fix reads the suspect file through a sandboxed filesystem MCP server, makes one LLM call for a minimal patch, validates it locally, writes it, and restarts the service — rolling back to the original bytes if the patched service won't come up.

Verify replays the exact failing request and demands exactly a 404, then re-queries SigNoz to confirm no new errors.

Total: 30 to 60 seconds, unattended.

📸 Screenshot: mission-control with the four agents mid-pipeline

The part that actually matters: the agents query SigNoz as a tool

This is the design decision I'd defend hardest.

SigNoz isn't where these agents report — it's where they perceive. Through the
SigNoz MCP server (which Foundry installs alongside SigNoz in one step), the agents call real tools:

signoz_search_traces — Monitor's detection loop, and Verify's post-fix check
signoz_search_logs — Diagnosis's primary evidence, because the stack trace lives in the log body
signoz_get_trace_details — the full span tree for the failing request

And because I wrapped every one of those calls in a span, the agents' use of SigNoz is itself observable in SigNoz. You can see how long each tool call took and whether it succeeded.

That recursion is the whole thesis of the project.

One incident = one trace

Two OTel services exist: demo-api (the app) and sdlc-agents (the swarm). A single incident produces one continuous trace of about 14 spans:

agent.orchestrator.incident              ← root: incident.id, phase events, span link
├── agent.diagnosis.run
│   ├── gen_ai.tool.signoz_search_logs
│   ├── gen_ai.tool.signoz_get_trace_details
│   └── agent.diagnosis.llm              ← model + input/output tokens
├── agent.fix.run
│   ├── gen_ai.tool.read_text_file
│   ├── agent.fix.llm
│   ├── gen_ai.tool.write_file
│   └── agent.fix.restart
└── agent.verify.run
    ├── agent.verify.replay              ← http.request.method, status 404
    └── gen_ai.tool.signoz_search_traces

📸 Screenshot: the SigNoz trace waterfall for one incident

A few conventions I locked down early and never broke:

Agent steps are agent.<name>.<step>. MCP tool calls are gen_ai.tool.<literal tool name>, always as a child of the step that invoked them. The gen_ai.tool.* namespace is strictly for MCP calls. When the Fix agent restarts a process or the Verify agent makes an HTTP request, those are steps — agent.fix.restart, agent.verify.replay — never dressed up as tool calls. Keeping that boundary clean is what makes the trace readable at a glance.

Every LLM call carries gen_ai.request.model, gen_ai.usage.input_tokens, and gen_ai.usage.output_tokens, per the OTel GenAI semantic conventions. Token spend per agent falls straight out of that.

incident.id is propagated through OTel Context, not function parameters. This one paid off. Threading an ID through every agent signature would have touched every function and silently missed any span whose caller forgot to pass it. Instead one Context key, read inside the two span-opening wrappers, stamps all 14 spans — including code I never modified. If you've used a ThreadLocal for a request ID in Java, it's exactly that, except AsyncLocalStorage keeps it correct across await.

The root span carries a span link to the Monitor poll that detected the incident — "caused by, not parented by." The poll is its own root trace; it existed before the incident did, so parenting would be a lie about causality.

Failure honesty, and why it's the best thing I built

Early on, a failed run just said "failed." Useless. I couldn't tell a bad hypothesis from an API outage.

So failures became explicit, on the span:

Reason	Meaning
`llm_error`	Model failed or returned junk, or a pre-write guard rejected the patch
`wrong_hypothesis`	Process is up but still misbehaving
`patch_broke_process`	Patch wouldn't boot; rollback succeeded
`rollback_failed`	Even rollback wouldn't boot — stop, alert a human
`telemetry_unavailable`	SigNoz/MCP is down — infrastructure, not reasoning
`fs_mcp_unavailable`	The filesystem MCP transport died

The rule that makes it work: a dead telemetry pipeline must never be reported as an agent reasoning badly. Those are completely different problems with completely different fixes, and collapsing them into "failed" destroys the only signal that matters.

There are also no retries anywhere. One attempt. If it fails, it fails loudly in the trace. Retries would have hidden exactly the failures I most wanted to see.

This paid for itself twice in one afternoon. Two runs failed; the spans told me instantly that one was a Gemini 503 (transient, provider-side) and the other was a telemetry ingest race (mine). Same symptom, completely different causes, zero debugging time.

SigNoz caught a bug in my own agent

My favourite moment.

The Verify agent waited a flat 5 seconds after the restart, then queried SigNoz once to confirm no new errors. A run failed with telemetry_unavailable.

The agent was right and my tuning was wrong. Spans take 5 to 10 seconds to become queryable after export. Verify was racing ingest, finding an empty window, and — correctly, by its own logic — reporting that the pipeline looked dead.

The fix was to separate waiting for data to arrive from asserting what the data says: poll the same fixed window every 3 seconds up to 21 seconds until it's non-empty, then evaluate the assertion exactly once. It's the difference between Thread.sleep(5000) before an assert and await().atMost(21, SECONDS).until(...).

I only found it because the failure was visible in the trace instead of swallowed by a retry.

Dashboards, and a lesson about metric labels

Two dashboards, version-controlled as JSON and provisioned idempotently through the SigNoz API:

Agent Fleet Health — per-agent invocations, p50/p95 latency, LLM token spend by agent, MCP tool-call breakdown, failure rate, and failures by reason.

Demo API Health — request volume, 5xx count, and a per-status-code breakdown of the failing route that visibly goes red at trigger and green after the fix.

Plus a threshold alert on agent failure rate.

📸 Screenshot: the Agent Fleet Health dashboard

The non-obvious part was which data source each panel needs. SigNoz's span metrics do cover internal spans — the span name shows up as an operation label, which I didn't expect. But they carry only service.name, operation, span.kind and status.code. Every custom attribute I'd carefully added — token counts, gen_ai.agent.name, incident.id, failure reason, even http.route — exists only
on spans.

So panels are split per-panel: latency and invocation counts come from metrics (cheap, pre-aggregated), while anything needing a custom dimension queries traces.

The general lesson: pre-aggregated metrics only carry the dimensions someone declared in advance. Assuming your custom span attributes are queryable as metric labels gets you panels that render empty rather than erroring — which is worse, because empty looks like "nothing happened."

Two more traps worth writing down:

signoz_calls_total is a monotonic delta counter, so it needs timeAggregation: "increase". The default rate silently returns nothing against sparse demo traffic.
Metric labels are dotted (service.name). The underscore form doesn't error — it returns empty group keys with a non-zero aggregate. Silently wrong.

And because "the API returned 200" proves nothing, every panel is validated by committed script that runs the panel's own query and fails any panel returning empty or all-zero data. I run it over a narrow 15-minute window containing only a fresh incident, so the panels have to populate from live data rather than accumulated history.

Does it generalise, or did I just special-case my own bug?

The obvious challenge. So I tested it.

I injected a completely different defect the system had never seen — a different route (GET /api/bugs, the list endpoint), a different error class (calling a method that doesn't exist rather than dereferencing null), at a different line:

const bugs = listBugs();

res.json(bugs);
res.json(bugs.sortByPriority());

No code changes. No prompt changes. I just broke it and watched.

Monitor detected it. Diagnosis reported src/index.ts:77 with the cause "attempts to invoke bugs.sortByPriority(), but sortByPriority is not a defined function on bugs" — exactly right. Fix repaired it. Verify confirmed it. Incident closed resolved.

Nothing about that bug was anticipated anywhere in the code.

Guardrails

An agent that rewrites source and restarts a live service needs hard limits.

Least privilege. The Fix agent reaches the codebase only through a filesystem MCP server rooted at the backend directory. It cannot resolve a path outside it — a committed test asserts that relative traversal, absolute paths, directory listing, and search are all refused.

One detail I nearly got wrong: the filesystem MCP server replaces its own allow-list with client-supplied roots if the client advertises the roots capability. My client declares no capabilities, deliberately. A server-side restriction is only as strong as the negotiation that can rewrite it.

Pre-write validation. A patch must match exactly once, must actually change something, and is capped at 300 characters per side and a 200-byte net delta. Violations abort before any write.

Automatic rollback. Original bytes captured before writing; restored if the patched service fails its health check. Mechanical reversion, not a second guess.

Things that cost me time

A model in ListModels is not a model you can call. gemini-2.5-flash appeared in the listing and returned 404 "no longer available to new users" on the actual call.
A listing endpoint is not an entitlement check — verify against your credentials.

Free-tier quota is per model, per day. 20 requests/day/model, and each incident spends 2.

Gemini flash models think by default, and thinking tokens come out of
maxOutputTokens. With a 1024-token budget, the entire allowance was consumed thinking before any JSON was emitted — finishReason: MAX_TOKENS, empty text. An empty response at MAX_TOKENS is a budget symptom, not a prompt-quality problem. Don't start by rewriting the prompt.

Authentication succeeding is not authorization working. A fresh SigNoz
service-account key returned 200 on /api/v1/version and on the identity endpoint while every data route returned 403 — because the account had no role attached. Test a privileged route, not a version route.

Instrumenting an ESM app needs more than --require. Node's ESM→CJS path bypasses require-in-the-middle, so a preload alone patched node:http but silently left Express unpatched. Spans still appeared — just named bare GET with no route. When verifying OTel setup, check span names and route attributes, not just span presence.

What I'd do next

Detection is error-shaped. Monitor triggers on 5xx and exception spans. A silent logic bug returning HTTP 200 with a wrong answer would never be noticed. Catching that needs assertions about correctness, not just errors.

Verify's replay is bug-specific. It re-requests the known failing URL. For the novel bug, the general safety net was the "zero error spans after restart" check. A more general Verify would derive its replay from the incident.

Closing

The self-healing loop is a good demo. But the thing I'd actually want in production is the boring part: when an agent fails, I want to know whether it was the agent's fault, and I want to answer that from a trace rather than from a hunch.

That means treating agents like services — spans for reasoning steps, child spans for tool calls, token usage as a first-class metric, and failure reasons specific enough to act on. SigNoz's MCP server made the interesting half possible, because the agents could consume observability rather than just emit it.

Repo: https://github.com/Nagarjuna325/Agent-Observability

Stack: TypeScript, OpenTelemetry, SigNoz OSS (deployed via Foundry), SigNoz MCP server, Google Gemini, Express, React. Demo app: BuggyBoard by Andrew Knight.

DEV Community: Nagarjuna Bolla