My AI agent was a black box. So we gave it eyes - meet AgentWatch

The 2 a.m. problem

A few weeks ago I was staring at a terminal at 2 a.m., watching an AI agent I had built do something. It was supposed to fix a broken test. Instead it just sat there, printing the odd line, quietly calling a model over and over.

Was it stuck? Was it working? How much had it spent? I had no idea. All I had was a wall of print() statements.

That was the moment it hit me: every other part of my stack has observability. My APIs, my databases, my queues all have dashboards, traces, and alerts. But my AI agents - the newest and most autonomous part of my system - were complete black boxes.

So my partner and I set out to fix that. Yes, I built this with an AI pair-programmer, which the hackathon happily allows. It felt only right to give an AI agent the tool it would want for itself.

We called it AgentWatch.

AgentWatch is a drop-in observability layer for autonomous AI agents. Add it in about 3 lines, and you see everything your agent does - every step, every token, every dollar - live in SigNoz.

The whole idea fits on a sticky note: if you can't observe your AI agents, you don't own them.

The core idea: an agent run is just a trace

Here's the insight the whole project is built on. When you squint at an autonomous agent, a single run looks exactly like a distributed trace:

• the top-level run is the root span
• every LLM call is a child span
• every tool call is a child span
• tokens and cost are just span attributes

OpenTelemetry already knows how to do traces. And the community recently agreed on GenAI semantic conventions - a standard way to record the model name, input tokens, output tokens, and cost. SigNoz reads those conventions natively.

So our job was simple: make it effortless to emit them. That's the @observe decorator.

from agentwatch import observe, tool_span

@observe(agent="e2e-test-fixer")     # that's basically the whole ask
def run(task):
    plan = llm.plan(task)            # auto-traced: model, tokens, cost
    with tool_span("run_tests"):     # auto-traced tool call
        run_the_tests()
    return plan

That's it. No glue code, no SigNoz-specific plumbing. Wrap your agent, and every run turns into a clean trace.

To prove it, we built an agent worth watching

A library is only as convincing as its demo. So we built a small but real autonomous agent: an E2E test-fixer. It uses Playwright to drive Chromium against a tiny throwaway shop app. The suite starts with three real bugs - a wrong selector, a missing wait, and a wrong assertion. The agent:

1. runs the tests and sees them fail,
2. asks a real LLM - Llama-3.3-70B via Groq - for the root cause,
3. patches the specs,
4. re-runs until everything is green.

Here it is doing exactly that:

Two attempts, all green. Nothing about that is faked - real browser, real test runs, real LLM calls.

Step 1: See everything - the trace

The moment that agent runs, its entire story shows up in SigNoz as one trace. You can see the whole reasoning chain: run, fail, three diagnose LLM calls, three patch steps, re-run, pass. And when you click any LLM span, you get the real GenAI attributes:

• model: llama-3.3-70b-versatile
• input tokens: 191, output tokens: 49
• cost: $0.000338
• finish reason: stop

For the first time, I could actually see what my agent was thinking and what each thought cost. That black box was finally open.

Step 2: Metrics, for free

Because AgentWatch speaks plain OpenTelemetry, SigNoz automatically derives full APM from the same traces - no extra work:

Latency percentiles, Apdex, error rate, and a per-operation breakdown - including a row for every model the agent used. We wrote zero extra instrumentation to get this.

Step 3: One dashboard for everything

AgentWatch also ships with a ready-made SigNoz dashboard you import in one click:

• Total runs: 178
• Fix success rate: 94.4%
• Total LLM cost: $3.15
• tokens by model, cost over time, and tool activity - all in one glance

The big idea: the agent watches - and stops - itself

This is the part I'm proudest of. Most observability is a one-way street: the agent emits data, and a human looks at a dashboard later. We asked a different question:

What if the agent could read its own telemetry, live, and make decisions with it?

So we closed the loop. AgentWatch can query the agent's own cost back from SigNoz at runtime and use it as a control signal - a cost circuit breaker. Before each cycle, the agent asks SigNoz how much it has spent, and if it is over budget, it stops itself.

cycle 1: ran fix_suite -> fixed=True in 2 attempts
   SigNoz read-back: spent=$0.0090 | success=100%
COST CIRCUIT BREAKER - SigNoz reports $0.0090 >= budget.
Agent halting itself before overspend.

Now SigNoz isn't just the observer - it's the safety mechanism. The loop is closed. That quietly-burn-money nightmare from 2 a.m. can't happen anymore, because the agent watches its own wallet.

Production-ready: alerts that actually fire

Watching is good. Getting paged is better. AgentWatch ships an alert pack that fires on cost spikes and failures - and yes, we verified it firing live in SigNoz:

Why we think this matters

Everyone is building AI agents right now. Almost no one can see what those agents are doing in production. AgentWatch is the small, reusable layer that fixes that - the SRE copilot layer for the agent era.

• Open standards, no lock-in - pure OpenTelemetry GenAI conventions.
• Drop-in - about 3 lines, works with OpenAI, Anthropic, Groq, or any OpenAI-compatible provider.
• Reusable - a real, pip-installable library, not a one-off script.
• Deep SigNoz integration - traces, metrics, logs, a shippable dashboard, firing alerts, and live read-back.

We tested it hard, too - the library has a passing test suite, including one nasty case that trips up most naive versions: keeping each concurrent run's tokens and cost correctly isolated.

What we learned

• Cost is a first-class signal, not an afterthought. The moment we put a dollar value on every LLM span, we started thinking about agents completely differently.
• Standards pay off fast. Because we emitted the GenAI conventions, SigNoz just understood our data.
• Observability can be a control plane. Reading telemetry back to govern the agent turned this from a nice wrapper into something we're genuinely excited about.

Try it

• Instrument your agent: install AgentWatch, add @observe, and point it at SigNoz.
• Run the demo with no keys, or drop in a free Groq key for real LLM calls.
• Code and setup: https://github.com/avinashjindal7866/agentwatch

The full demo

Here's the whole thing, start to finish - the agent, the trace, the metrics, the dashboard, the self-governing loop, and the alert:

If you can't observe your AI agents, you don't own them. Now, finally, we do.

Built for the Agents of SigNoz hackathon.