My AI agent didn't crash. It developed OCD.

Let me be honest about where this started: I did not know much about observability. Spans, traces, exporters, OTLP endpoints, these were words other people used at work. I build agents, and when mine broke, my debugging strategy was reading logs and guessing.

Then the Agents of SigNoz hackathon showed up, and I signed up anyway, on the theory that a deadline is the fastest teacher. This is the story of going from "what exactly is a span?" to shipping a working psychiatric hospital for AI agents in a couple of weeks, paired with Claude Code the whole way. I'll be honest about which parts were me and which were the machine, because that split was the most interesting part of the project.

It began with a trace I couldn't stop staring at. Latency normal, error rate zero, every span green. The agent had called web_search("confirm pincode 110011") eighteen times with identical arguments, produced nothing, and by every dashboard I had, it was healthy.

Agents rarely fail the way services fail. They don't 500. They go insane: quietly, expensively, with a perfectly green dashboard. So I built them a hospital: AIIMS, the All India Institute of Machine Sanity. Sick agents get admitted, diagnosed, and treated. The joke was my starting point; making it sit on real, detectable failures took everything I had.

Setting a trap for my own agents

My first week was remedial: reading OpenTelemetry docs, asking Claude beginner questions I was too embarrassed to ask an SRE. The decision I'm glad I made early: instead of instrumenting one real app and hoping it misbehaved, I built five small tool-using agents, each engineered to fail one way, and shipped their traces to SigNoz with GenAI semantic conventions on the attributes.

The five failure modes, all of which I've hit for real in agent code:

1. Looping on an identical tool call until something kills the process
2. Retrying a dead endpoint in a tightening storm
3. Inflating context until the model forgets the task and answers a different question
4. Stalling mid-run, holding a lock, alive and doing nothing
5. Crashing outright, the only one a normal dashboard actually catches

Ordinary panels missed cases 1 to 4: p95 latency fine, error rate near zero. Case 3 is nastiest, the run succeeds, returns a confident answer to a question nobody asked. Green dashboard, broken product. The signal was in the traces the whole time; eighteen spans sharing a tool name and an argument hash isn't subtle once you look, but looking required already suspecting it. So instead of spotting these by eye, I wrote detectors.

Writing a detector that reads traces, not metrics

Here the pairing rhythm settled in: I'd describe a pathology, Claude would draft the detector, and I'd poke holes in it, usually by feeding it a healthy trace and watching it panic. Each pathology is one pure function over a list of span dicts, testable without SigNoz, an agent, or an LLM in the loop.

The loop detector is the simplest and caught the most real bugs:

def detect_chakravyuh(spans):
    """The same tool called with the same arguments, over and over."""
    counts = {}
    for s in _tool_spans(spans):
        sig = (s["attributes"]["gen_ai.tool.name"],
               s["attributes"].get("gen_ai.tool.args_hash", ""))
        counts[sig] = counts.get(sig, 0) + 1
    _, repeats = max(counts.items(), key=lambda kv: kv[1])
    if repeats < REPEAT_THRESHOLD:
        return None
    ...

The load-bearing detail is args_hash, a stable hash of the tool arguments emitted as a span attribute at call time:

def args_hash(args: dict) -> str:
    blob = json.dumps(args, sort_keys=True, default=str)
    return hashlib.sha1(blob.encode()).hexdigest()[:12]

sort_keys=True is not optional. Without it, {"a":1,"b":2} and {"b":2,"a":1} hash differently and the loop becomes invisible. I lost twenty minutes to that: the bug wasn't in anything clever, just a default argument I didn't know existed. Hashing instead of storing raw arguments also solved a real problem: arguments can carry user data you don't want in telemetry, and a 12-character hash groups perfectly in SigNoz while leaking nothing.

The context-drift detector was the one I expected to need embeddings for. A Jaccard overlap of content words between the run's stated goal and the current task, combined with context-window pressure, catches the real cases. Neither signal alone is enough: high pressure while still on-task is just a long run, low overlap early on is just a subtask. It's the conjunction, "the window filled up and the goal fell out of it," that was the single most useful modelling insight in the project. Every detector ships with two tests, one proving it fires and one proving it stays quiet on healthy runs. A diagnostic panel that diagnoses everyone is not a diagnostic panel.

Feeding the diagnosis back into SigNoz

The detectors produce a 0 to 100 sanity score per run. My first instinct was to show it in my own UI, the wrong instinct. Pushing it into SigNoz as a real metric was the best architectural call in the project:

sanity = meter.create_gauge("aiims.patient.sanity_score")
sanity.set(score, {"agent.name": name, "agent.run_id": run_id})

Because it sits next to the traces it was derived from, I get dashboard panels, correlation with raw spans, and, the actual point, alerting: a threshold rule on sanity_score < 35 pages me when an agent is losing its mind, which no latency alert ever would. One gotcha that cost me an evening: the exporter needs the signal path appended to the SigNoz endpoint (/v1/traces), and the failure mode is silence, not an error.

Treating the patient

Once the alert fired reliably, the obvious question was: why am I in this loop at all? By the time I read it, the money is already spent. So the same detectors now run in-process, during the run. When a disorder crosses a threshold, the supervisor changes what the agent can do next: refuses the repeated call, trips a circuit breaker on a dead tool, enforces a token budget, re-injects the original goal. Every intervention is itself emitted as a span, so I can prove a loop was broken instead of just claiming it.

Then I ran each agent twice, changing only whether the supervisor was attached: same agents, same scripted pathologies, same goals.

  patient      disorder   arm         steps     tokens  outcome
  abhimanyu    loop       untreated      20    101,954  survived
                          treated        11     22,208  survived
  damini       stall      untreated       2      2,060  DIED
                          treated         4      4,580  survived

  TOTAL   tokens 1,497,222 → 582,499 (61% saved), deaths 1 → 0

Note damini spends more tokens under treatment, which is correct, since she finishes the job instead of dying halfway. I nearly "fixed" that row before realising completed runs, not spend, was the metric I cared about.

What I got wrong

A rule I kept from day one: nothing Claude produced was accepted on its word, every finding got checked against live SigNoz first. That's the only reason these are a blog section instead of surprises in front of judges.

A therapeutic abort is not a crash. When the budget guard killed a runaway run, my own death detector filed a death certificate for it, so the system punished itself for working. The fix was an attribute the detector skips on deliberate stops. Stops and failures look identical in span status; only you know the difference, so you have to record it.

One global tracer provider will silently lie to you. Running several agents in one process, I called set_tracer_provider() per agent. OpenTelemetry logs a warning, not an error, and quietly keeps the first provider. Every agent after the first had an empty span buffer, so every diagnosis came back "perfectly healthy." My comparison table was garbage for an hour. Thresholds are the whole game beyond that: my first loop detector fired at 2 repeats and flagged every agent that retried anything, until I tuned it against agents deliberately broken one specific way each.

Who actually built this

"Built with AI" can mean "I typed one prompt" or "I did everything and mentioned AI for the algorithm." Neither is true here. The idea and framing were mine, the hospital, the disorder taxonomy, the conviction that a diagnosis which doesn't change an outcome is decoration, so the ward had to treat, not just report, and every architecture fork was my call. Claude Code wrote most of the actual lines: detectors, supervisor, ward service, tests, UI. I directed, reviewed, rejected, verified. Three weeks ago that split would have felt like cheating; now it's just what building looks like. Taste and accountability stayed human, the typing largely didn't.

Takeaway

Agent failures are behavioural, not statistical: the spans are all successful and the shape between them is what's wrong. Traces already contain that shape. Write detectors over span data, push the result back as a metric so it can page you, and once it can page you, ask why a human is in the loop at all.

If you're staring at a field you know nothing about: I could not define "span" when this started. Pick a deadline, bring a pair partner that never tires of your questions, and refuse to accept anything you haven't verified yourself.

Code, detectors and dashboards: github.com/githubber-me/aiims

Built for the Agents of SigNoz hackathon, following the OpenTelemetry GenAI semantic conventions.