I Built Four AI Agents That Fix Bugs by Themselves, Then Made Them Watch Themselves Work

We've gotten good at observing software. We're not yet good at observing the agents
that increasingly write and operate that software.

When a service throws a 500, you open a trace and see exactly which span failed. When
an AI agent does something wrong, you usually get… a log line saying it failed. Maybe
a stack trace. Rarely the reasoning, the tool calls it made, what it cost, or — most
importantly — whether the failure was the agent's fault at all.

So for the Agents of SigNoz hackathon I built AgentOps: four autonomous agents
that detect, diagnose, fix, and verify a real bug in a real application with no human
in the loop. The self-healing part is the demo. The actual project is that every
thought those agents have is a span in SigNoz.

---

The setup

I deliberately did not scaffold my own toy app to break. Fixing a bug you planted in
code you wrote is not convincing.

Instead I vendored BuggyBoard,
an MIT-licensed bug tracker by Andrew Knight — Express, TypeScript, SQLite. Foreign
code, with a commit history that predates this project. Then I seeded exactly one
defect into its not-found guard:

  const bug = getBug(id);
- if (!bug) {
+ if (!bug.id) {
    res.status(404).json({ error: "not_found", message: "Bug not found." });

Now GET /api/bugs/999999 dereferences null, throws a TypeError, and Express
returns 500 where it should return 404. A boring, extremely realistic bug.

There's a nice symmetry to it: the agents fix a bug in a bug tracker.

---

The loop

Monitor  →  Diagnosis  →  Fix  →  Verify

Monitor polls SigNoz every four seconds asking "any error spans from demo-api
since my last check?" It has no privileged backchannel to the application. It
finds out the same way a human on-call engineer would — by querying the observability
backend. It makes zero LLM calls; the threshold is deterministic. Its agenthood is the
autonomous loop, not cleverness.

Diagnosis pulls the ERROR logs and the full trace from SigNoz, hands only that
evidence to an LLM, and demands strict JSON back: file, line, cause. It answers
src/index.ts:87. It never reads the source code, and it certainly never reads the
answer key.

Fix reads the suspect file through a sandboxed filesystem MCP server, makes one
LLM call for a minimal patch, validates it locally, writes it, and restarts the
service — rolling back to the original bytes if the patched service won't come up.

Verify replays the exact failing request and demands exactly a 404, then
re-queries SigNoz to confirm no new errors.

Total: 30 to 60 seconds, unattended.

📸 Screenshot: mission-control with the four agents mid-pipeline

---

The part that actually matters: the agents query SigNoz as a tool

This is the design decision I'd defend hardest.

SigNoz isn't where these agents report — it's where they perceive. Through the
SigNoz MCP server (which Foundry installs alongside SigNoz in one step), the
agents call real tools:

• signoz_search_traces — Monitor's detection loop, and Verify's post-fix check
• signoz_search_logs — Diagnosis's primary evidence, because the stack trace lives in the log body
• signoz_get_trace_details — the full span tree for the failing request

And because I wrapped every one of those calls in a span, the agents' use of SigNoz
is itself observable in SigNoz. You can see how long each tool call took and whether
it succeeded.

That recursion is the whole thesis of the project.

---

One incident = one trace

Two OTel services exist: demo-api (the app) and sdlc-agents (the swarm). A single
incident produces one continuous trace of about 14 spans:

agent.orchestrator.incident              ← root: incident.id, phase events, span link
├── agent.diagnosis.run
│   ├── gen_ai.tool.signoz_search_logs
│   ├── gen_ai.tool.signoz_get_trace_details
│   └── agent.diagnosis.llm              ← model + input/output tokens
├── agent.fix.run
│   ├── gen_ai.tool.read_text_file
│   ├── agent.fix.llm
│   ├── gen_ai.tool.write_file
│   └── agent.fix.restart
└── agent.verify.run
    ├── agent.verify.replay              ← http.request.method, status 404
    └── gen_ai.tool.signoz_search_traces

📸 Screenshot: the SigNoz trace waterfall for one incident

A few conventions I locked down early and never broke:

Agent steps are agent.<name>.<step>. MCP tool calls are gen_ai.tool.<literal tool
name>, always as a child of the step that invoked them. The gen_ai.tool.*
namespace is strictly for MCP calls. When the Fix agent restarts a process or the
Verify agent makes an HTTP request, those are steps — agent.fix.restart,
agent.verify.replay — never dressed up as tool calls. Keeping that boundary clean is
what makes the trace readable at a glance.

Every LLM call carries gen_ai.request.model, gen_ai.usage.input_tokens, and
gen_ai.usage.output_tokens, per the OTel GenAI semantic conventions. Token spend
per agent falls straight out of that.

incident.id is propagated through OTel Context, not function parameters. This one
paid off. Threading an ID through every agent signature would have touched every
function and silently missed any span whose caller forgot to pass it. Instead one
Context key, read inside the two span-opening wrappers, stamps all 14 spans —
including code I never modified. If you've used a ThreadLocal for a request ID in
Java, it's exactly that, except AsyncLocalStorage keeps it correct across await.

The root span carries a span link to the Monitor poll that detected the
incident — "caused by, not parented by." The poll is its own root trace; it existed
before the incident did, so parenting would be a lie about causality.

---

Failure honesty, and why it's the best thing I built

Early on, a failed run just said "failed." Useless. I couldn't tell a bad hypothesis
from an API outage.

So failures became explicit, on the span:

Reason	Meaning
llm_error	Model failed or returned junk, or a pre-write guard rejected the patch
wrong_hypothesis	Process is up but still misbehaving
patch_broke_process	Patch wouldn't boot; rollback succeeded
rollback_failed	Even rollback wouldn't boot — stop, alert a human
telemetry_unavailable	SigNoz/MCP is down — infrastructure, not reasoning
fs_mcp_unavailable	The filesystem MCP transport died

The rule that makes it work: a dead telemetry pipeline must never be reported as an
agent reasoning badly. Those are completely different problems with completely
different fixes, and collapsing them into "failed" destroys the only signal that
matters.

There are also no retries anywhere. One attempt. If it fails, it fails loudly in
the trace. Retries would have hidden exactly the failures I most wanted to see.

This paid for itself twice in one afternoon. Two runs failed; the spans told me
instantly that one was a Gemini 503 (transient, provider-side) and the other was a
telemetry ingest race (mine). Same symptom, completely different causes, zero
debugging time.

---

SigNoz caught a bug in my own agent

My favourite moment.

The Verify agent waited a flat 5 seconds after the restart, then queried SigNoz once
to confirm no new errors. A run failed with telemetry_unavailable.

The agent was right and my tuning was wrong. Spans take 5 to 10 seconds to become
queryable after export. Verify was racing ingest, finding an empty window, and —
correctly, by its own logic — reporting that the pipeline looked dead.

The fix was to separate waiting for data to arrive from asserting what the data
says: poll the same fixed window every 3 seconds up to 21 seconds until it's
non-empty, then evaluate the assertion exactly once. It's the difference between
Thread.sleep(5000) before an assert and await().atMost(21, SECONDS).until(...).

I only found it because the failure was visible in the trace instead of swallowed by
a retry.

---

Dashboards, and a lesson about metric labels

Two dashboards, version-controlled as JSON and provisioned idempotently through the
SigNoz API:

Agent Fleet Health — per-agent invocations, p50/p95 latency, LLM token spend by
agent, MCP tool-call breakdown, failure rate, and failures by reason.

Demo API Health — request volume, 5xx count, and a per-status-code breakdown of
the failing route that visibly goes red at trigger and green after the fix.

Plus a threshold alert on agent failure rate.

📸 Screenshot: the Agent Fleet Health dashboard

The non-obvious part was which data source each panel needs. SigNoz's span metrics
do cover internal spans — the span name shows up as an operation label, which I
didn't expect. But they carry only service.name, operation, span.kind and
status.code. Every custom attribute I'd carefully added — token counts,
gen_ai.agent.name, incident.id, failure reason, even http.route — exists only
on spans.

So panels are split per-panel: latency and invocation counts come from metrics
(cheap, pre-aggregated), while anything needing a custom dimension queries traces.

The general lesson: pre-aggregated metrics only carry the dimensions someone declared
in advance. Assuming your custom span attributes are queryable as metric labels gets
you panels that render empty rather than erroring — which is worse, because empty
looks like "nothing happened."

Two more traps worth writing down:

• signoz_calls_total is a monotonic delta counter, so it needs timeAggregation: "increase". The default rate silently returns nothing against sparse demo traffic.
• Metric labels are dotted (service.name). The underscore form doesn't error — it returns empty group keys with a non-zero aggregate. Silently wrong.

And because "the API returned 200" proves nothing, every panel is validated by a
committed script that runs the panel's own query and fails any panel returning
empty or all-zero data. I run it over a narrow 15-minute window containing only a
fresh incident, so the panels have to populate from live data rather than accumulated
history. 16/16.

---

Does it generalise, or did I just special-case my own bug?

The obvious challenge. So I tested it.

I injected a completely different defect the system had never seen — a different
route (GET /api/bugs, the list endpoint), a different error class (calling a method
that doesn't exist rather than dereferencing null), at a different line:

  const bugs = listBugs();
- res.json(bugs);
+ res.json(bugs.sortByPriority());

No code changes. No prompt changes. I just broke it and watched.

Monitor detected it. Diagnosis reported src/index.ts:77 with the cause
"attempts to invoke bugs.sortByPriority(), but sortByPriority is not a defined
function on bugs" — exactly right. Fix repaired it. Verify confirmed it. Incident
closed resolved.

Nothing about that bug was anticipated anywhere in the code.

---

Guardrails

An agent that rewrites source and restarts a live service needs hard limits.

Least privilege. The Fix agent reaches the codebase only through a filesystem MCP
server rooted at the backend directory. It cannot resolve a path outside it — a
committed test asserts that relative traversal, absolute paths, directory listing, and
search are all refused.

One detail I nearly got wrong: the filesystem MCP server replaces its own allow-list
with client-supplied roots if the client advertises the roots capability. My client
declares no capabilities, deliberately. A server-side restriction is only as strong as
the negotiation that can rewrite it.

Pre-write validation. A patch must match exactly once, must actually change
something, and is capped at 300 characters per side and a 200-byte net delta.
Violations abort before any write.

Automatic rollback. Original bytes captured before writing; restored if the patched
service fails its health check. Mechanical reversion, not a second guess.

---

Things that cost me time

A model in ListModels is not a model you can call. gemini-2.5-flash appeared in
the listing and returned 404 "no longer available to new users" on the actual call.
A listing endpoint is not an entitlement check — verify against your credentials.

Free-tier quota is per model, per day. 20 requests/day/model, and each incident
spends 2. A day of rehearsals exhausted one model. Because the quota is per-model,
switching IDs buys another 20 — so the model became an environment variable.

Gemini flash models think by default, and thinking tokens come out of
maxOutputTokens. With a 1024-token budget, the entire allowance was consumed
thinking before any JSON was emitted — finishReason: MAX_TOKENS, empty text. An
empty response at MAX_TOKENS is a budget symptom, not a prompt-quality problem. Don't
start by rewriting the prompt.

Authentication succeeding is not authorization working. A fresh SigNoz
service-account key returned 200 on /api/v1/version and on the identity endpoint
while every data route returned 403 — because the account had no role attached. Test a
privileged route, not a version route.

Instrumenting an ESM app needs more than --require. Node's ESM→CJS path bypasses
require-in-the-middle, so a preload alone patched node:http but silently left Express
unpatched. Spans still appeared — just named bare GET with no route. When verifying
OTel setup, check span names and route attributes, not just span presence.

---

What I'd do next

Detection is error-shaped. Monitor triggers on 5xx and exception spans. A silent
logic bug returning HTTP 200 with a wrong answer would never be noticed. Catching that
needs assertions about correctness, not just errors.

Verify's replay is bug-specific. It re-requests the known failing URL. For the
novel bug, the general safety net was the "zero error spans after restart" check. A
more general Verify would derive its replay from the incident.

---

Closing

The self-healing loop is a good demo. But the thing I'd actually want in production is
the boring part: when an agent fails, I want to know whether it was the agent's
fault, and I want to answer that from a trace rather than from a hunch.

That means treating agents like services — spans for reasoning steps, child spans for
tool calls, token usage as a first-class metric, and failure reasons specific enough
to act on. SigNoz's MCP server made the interesting half possible, because the agents
could consume observability rather than just emit it.

Repo: https://github.com/Nagarjuna325/Agent-Observability

Stack: TypeScript, OpenTelemetry, SigNoz OSS (deployed via Foundry), SigNoz MCP
server, Google Gemini, Express, React. Demo app: BuggyBoard by Andrew Knight (MIT).