Building AgentLens: An Autonomous, Self-Healing AI SRE Copilot with SigNoz

The Problem: Generative AI is a Financial Black Box

Generative AI agents are very powerful but they are also a mystery when it comes to money and how they work. If something goes wrong with an agent like it gets stuck in a loop it can spend all of a startups money for models in one night. Most tools that are supposed to help us see what is going on with AI just show us what already happened and then people have to fix the problem.

I wanted to make something that would really solve this problem. For the WeMakeDevs SigNoz Hackathon I made AgentLens: a tool that helps AI agents work better by watching what they do. AgentLens uses SigNozs OpenTelemetry traces and alerts to tell the agent what to do. If the agent starts to cost too much money or does not work well it can fix itself by switching to a cheaper model or stopping before a person even notices.

Here is how I made it using LangGraph, FastAPI and SigNoz including the code I wrote and the problems I had when I was testing it.

Step 1: Deep OpenTelemetry Instrumentation

Before an agent can fix itself it needs to know what it is doing. I made the part of the agent using LangGraph, which is like a plan that the agent follows. To get information into SigNoz I set up the OpenTelemetry Python SDK in my otel_setup.py file. I made a TracerProvider and a MeterProvider with settings for AI costs.

The important part was adding special details to the agents actions, in the agent.py file. For example when the agent was generating text I tracked how many tokens it used and calculated how much it cost in real money. I made a graph to show this information, which is called a histogram metric and it is named gen_ai.usage.cost_usd.

`def _record_llm_cost(span, model: str, usage: dict, session_id: str):
meter = _get_meter()
cost_hist = meter.create_histogram("gen_ai.usage.cost_usd", unit="usd")

input_tokens = usage.get("prompt_tokens", 0) if usage else 0
output_tokens = usage.get("completion_tokens", 0) if usage else 0
cost = _cost_usd(model, input_tokens, output_tokens)

span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
span.set_attribute("gen_ai.usage.cost_usd", cost)
cost_hist.record(cost, {"model": model, "session_id": session_id})

return cost`

By pushing this via OTLP to my local SigNoz instance, I could instantly see the exact waterfall of my agent's thought process.

Step 2: Closing the Loop with Webhooks & Self-Healing
Passive observability isn't enough. I set up two alerts in SigNoz: Cost Spike Per Session and Hallucination Score Drop. I configured these alerts to hit a local webhook endpoint.

When SigNoz detects a spike, it POSTs to the webhook, which triggers my self_healing.py controller. This flips an in-memory flag. The next time the agent runs a node, it checks these flags. If the cost guard is active, it silently swaps the expensive model (llama-3.3-70b-versatile) for the much cheaper llama-3.1-8b-instant:

# From agent.py inside generate_node
model_to_use = AGENT_MODEL
if self_healing.get_state().cost_alert_active:
    model_to_use = JUDGE_MODEL  # cheaper model
    self_healing.annotate_span(span, "downgraded_to_cheap_model")
    self_healing.record_action()
    span.add_event(
        "self_healing_applied",
        {"action": "downgraded_to_cheap_model", "model": model_to_use},
    )

This forces the agent to autonomously adapt mid-session, recording the exact action on the trace so SREs can see the self-healing event in SigNoz.

Step 3: Natural-Language Diagnostics with SigNoz MCP
I wanted AgentLens to explain why a request failed in plain English. In mcp_diagnostic.py, I connected to the signoz-mcp-server as a client to pull live span data via the signoz_get_trace_details tool.

What I learned (The debugging story): When I first ran this, my LLM crashed with a 413 Request too large error. It turns out the raw OpenTelemetry trace JSON returned by the MCP server is massive, and my free-tier Groq API key has a strict 6,000 Tokens Per Minute limit!

I had to write a custom summarization function (_summarize_trace) to parse the SigNoz query-range response, sort the spans by duration, and extract only the most important attributes before sending it to the LLM.

`def _summarize_trace(trace_data_text: str) -> str:
# ... JSON parsing ...
rows_sorted = sorted(rows, key=_row_duration_ns, reverse=True)[:MAX_SPANS_TO_INCLUDE]
lines = [f"Top {len(rows_sorted)} spans by duration:"]

for row in rows_sorted:
    duration_ms = _row_duration_ns(row) / 1_000_000
    name = _row_name(row)

    # Only pull relevant AI and HTTP attributes to save tokens
    interesting = {
        k: v for k, v in row.items()
        if isinstance(k, str)
        and k.startswith(("gen_ai.", "agentlens.", "http.", "rpc."))
        and v not in (None, "", [], {})
    }

    line = f"- {name}: {duration_ms:.1f}ms | {json.dumps(interesting)}"
    lines.append(line)

# Hard cap at ~750 tokens to guarantee we never crash the LLM
return "\n".join(lines)[:MAX_PROMPT_CHARS]`

By defensively truncating the prompt and filtering for keys starting with gen_ai. or agentlens., the LLM could successfully diagnose bottlenecks without blowing past token limits.

Outcomes and What I’d Do Differently
The outcome was a fully functioning SRE Sidekick that successfully catches cost explosions (simulated via my /chaos endpoint) and applies self-healing.

If I were to build this again, I would focus on improving the MCP trace summarization. Currently, the diagnosis parses spans in a flat list sorted by duration. What I'd do differently next time is parse the parent-child span trees, allowing the LLM to distinguish between a parent span being inherently slow versus it just waiting on a bottlenecked child span.

Conclusion

Building AgentLens showed that observability tools do not have to be passive. By instrumenting your code with OpenTelemetry and connecting SigNoz alerts directly back, into your applications logic you can create systems that do not just alert you when they break on a dashboard. These systems actively heal themselves in time.