DEV Community: vishruthvayu

Building AgentLens: An Autonomous, Self-Healing AI SRE Copilot with SigNoz

vishruthvayu — Sun, 26 Jul 2026 11:32:14 +0000

The Problem: Generative AI is a Financial Black Box

Generative AI agents are very powerful but they are also a mystery when it comes to money and how they work. If something goes wrong with an agent like it gets stuck in a loop it can spend all of a startups money for models in one night. Most tools that are supposed to help us see what is going on with AI just show us what already happened and then people have to fix the problem.

I wanted to make something that would really solve this problem. For the WeMakeDevs SigNoz Hackathon I made AgentLens: a tool that helps AI agents work better by watching what they do. AgentLens uses SigNozs OpenTelemetry traces and alerts to tell the agent what to do. If the agent starts to cost too much money or does not work well it can fix itself by switching to a cheaper model or stopping before a person even notices.

Here is how I made it using LangGraph, FastAPI and SigNoz including the code I wrote and the problems I had when I was testing it.

Step 1: Deep OpenTelemetry Instrumentation

Before an agent can fix itself it needs to know what it is doing. I made the part of the agent using LangGraph, which is like a plan that the agent follows. To get information into SigNoz I set up the OpenTelemetry Python SDK in my otel_setup.py file. I made a TracerProvider and a MeterProvider with settings for AI costs.

The important part was adding special details to the agents actions, in the agent.py file. For example when the agent was generating text I tracked how many tokens it used and calculated how much it cost in real money. I made a graph to show this information, which is called a histogram metric and it is named gen_ai.usage.cost_usd.

`def _record_llm_cost(span, model: str, usage: dict, session_id: str):
meter = _get_meter()
cost_hist = meter.create_histogram("gen_ai.usage.cost_usd", unit="usd")

input_tokens = usage.get("prompt_tokens", 0) if usage else 0
output_tokens = usage.get("completion_tokens", 0) if usage else 0
cost = _cost_usd(model, input_tokens, output_tokens)

span.set_attribute("gen_ai.request.model", model)
span.set_attribute("gen_ai.usage.input_tokens", input_tokens)
span.set_attribute("gen_ai.usage.output_tokens", output_tokens)
span.set_attribute("gen_ai.usage.cost_usd", cost)
cost_hist.record(cost, {"model": model, "session_id": session_id})

return cost`

By pushing this via OTLP to my local SigNoz instance, I could instantly see the exact waterfall of my agent's thought process.

Step 2: Closing the Loop with Webhooks & Self-Healing
Passive observability isn't enough. I set up two alerts in SigNoz: Cost Spike Per Session and Hallucination Score Drop. I configured these alerts to hit a local webhook endpoint.

When SigNoz detects a spike, it POSTs to the webhook, which triggers my self_healing.py controller. This flips an in-memory flag. The next time the agent runs a node, it checks these flags. If the cost guard is active, it silently swaps the expensive model (llama-3.3-70b-versatile) for the much cheaper llama-3.1-8b-instant:

# From agent.py inside generate_node
model_to_use = AGENT_MODEL
if self_healing.get_state().cost_alert_active:
    model_to_use = JUDGE_MODEL  # cheaper model
    self_healing.annotate_span(span, "downgraded_to_cheap_model")
    self_healing.record_action()
    span.add_event(
        "self_healing_applied",
        {"action": "downgraded_to_cheap_model", "model": model_to_use},
    )

This forces the agent to autonomously adapt mid-session, recording the exact action on the trace so SREs can see the self-healing event in SigNoz.

Step 3: Natural-Language Diagnostics with SigNoz MCP
I wanted AgentLens to explain why a request failed in plain English. In mcp_diagnostic.py, I connected to the signoz-mcp-server as a client to pull live span data via the signoz_get_trace_details tool.

What I learned (The debugging story): When I first ran this, my LLM crashed with a 413 Request too large error. It turns out the raw OpenTelemetry trace JSON returned by the MCP server is massive, and my free-tier Groq API key has a strict 6,000 Tokens Per Minute limit!

I had to write a custom summarization function (_summarize_trace) to parse the SigNoz query-range response, sort the spans by duration, and extract only the most important attributes before sending it to the LLM.

`def _summarize_trace(trace_data_text: str) -> str:
# ... JSON parsing ...
rows_sorted = sorted(rows, key=_row_duration_ns, reverse=True)[:MAX_SPANS_TO_INCLUDE]
lines = [f"Top {len(rows_sorted)} spans by duration:"]

for row in rows_sorted:
    duration_ms = _row_duration_ns(row) / 1_000_000
    name = _row_name(row)

    # Only pull relevant AI and HTTP attributes to save tokens
    interesting = {
        k: v for k, v in row.items()
        if isinstance(k, str)
        and k.startswith(("gen_ai.", "agentlens.", "http.", "rpc."))
        and v not in (None, "", [], {})
    }

    line = f"- {name}: {duration_ms:.1f}ms | {json.dumps(interesting)}"
    lines.append(line)

# Hard cap at ~750 tokens to guarantee we never crash the LLM
return "\n".join(lines)[:MAX_PROMPT_CHARS]`

By defensively truncating the prompt and filtering for keys starting with gen_ai. or agentlens., the LLM could successfully diagnose bottlenecks without blowing past token limits.

Outcomes and What I’d Do Differently
The outcome was a fully functioning SRE Sidekick that successfully catches cost explosions (simulated via my /chaos endpoint) and applies self-healing.

If I were to build this again, I would focus on improving the MCP trace summarization. Currently, the diagnosis parses spans in a flat list sorted by duration. What I'd do differently next time is parse the parent-child span trees, allowing the LLM to distinguish between a parent span being inherently slow versus it just waiting on a bottlenecked child span.

Conclusion

Building AgentLens showed that observability tools do not have to be passive. By instrumenting your code with OpenTelemetry and connecting SigNoz alerts directly back, into your applications logic you can create systems that do not just alert you when they break on a dashboard. These systems actively heal themselves in time.

Debugging the Black Box: My First Week With SigNoz

vishruthvayu — Fri, 17 Jul 2026 17:55:32 +0000

I spent weeks building a 4-step LLM agent (plan → retrieve → generate → judge) and then realized I had zero visibility into why it was failing or costing money. I had been building this agent for a little while before I really understood why "just add logging" doesn't cut it. When something went wrong, I had no way to answer the most basic question: which step actually broke? Was the LLM slow, or was my retrieval step the bottleneck? Was I paying for one LLM call or six because of a silent retry loop? I was flying blind in exactly the way observability tools exist to fix; I just hadn't set one up yet.

So, before starting my actual hackathon build, I spent an afternoon self-hosting SigNoz and wiring it into a small FastAPI service, mostly to answer one question: is this actually going to help me see inside my agent, or is it just another dashboard I'll ignore?

Setting it up (and hitting the first wall immediately)

I went with self-hosted SigNoz—no cloud dependency, full control, and I wanted to be sure I could reproduce my own setup later. My first mistake was following an old mental model of the install process. As of v0.130.0, the old docker-compose.yaml paths are deprecated in favor of a new CLI called Foundry.

The actual process is now much cleaner:

curl -fsSL [https://signoz.io/foundry.sh](https://signoz.io/foundry.sh) | sh

cat > casting.yaml <<'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
EOF

foundryctl cast -f casting.yaml

A couple minutes later: five containers were up, and the UI was live at localhost:8080.

Wiring up the first trace
Instrumenting a FastAPI service turned out to be genuinely three pieces of code, not the sprawling configuration I expected:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(tracer_provider)
FastAPIInstrumentor.instrument_app(app)

Every request my app handled immediately showed up as a trace, with zero manual span creation for the HTTP layer. That part was easy. The part that took real debugging was getting cost to show up as a first-class metric, not just a log line.

Where it actually got interesting: custom attributes
Auto-instrumentation gives you HTTP-level visibility for free, but it doesn't know about LLM costs. The moment I cared about was adding a manual span around my agent's decision logic and attaching custom attributes:

with tracer.start_as_current_span("agent.generate") as span:
    span.set_attribute("gen_ai.usage.cost_usd", cost)
    span.set_attribute("gen_ai.usage.input_tokens", usage.input_tokens)
    span.set_attribute("gen_ai.request.model", model_name)

Caption: Full trace view with custom gen_ai attributes and hallucination score.

Watching gen_ai.usage.cost_usd show up as a searchable, filterable field on the span—not buried in a log string—was the moment this stopped feeling like logging and started feeling like observability. I could click into any single request and see exactly what it cost, which model handled it, and how long each step took, all correlated by one trace_id.

The dumb bug that taught me the most
While building a dashboard panel to chart that cost metric over time, I hit something that looked broken but wasn't. My cost values are tiny—around $0.0001 per request—and SigNoz's panel decimal precision only goes up to three decimal places. 0.0001 rounded to 0.000 on the chart, so the line just sat flat at zero. The fix wasn't a SigNoz bug, it was a unit-scaling problem: I added a formula (A * 1000000) to the query builder to convert the metric into millionths of a dollar before charting it.

Caption: SigNoz Time Series showing agent step latencies.

I also learned that similarity_search_with_relevance_scores in my retriever doesn't always return values between 0 and 1; I saw negative numbers in my logs. It was harmless for my use case, but it served as a reminder to actually read library warnings instead of assuming output is always well-behaved.

Setting Up Alerts for the Ops Console & Self-Healing
This initial exploration was just the warm-up. For the main hackathon build, I focused on building a highly reactive self-healing loop. I created a custom alert rule targeting the gen_ai.usage.cost_usd attribute. When the cost breached the threshold, SigNoz triggers a webhook to my FastAPI backend.

@app.post("/webhooks/signoz-alert")
async def signoz_alert_webhook(request: Request):
    payload = await request.json()
    alerts = payload.get("alerts") or [payload]
    triggered = []
    for a in alerts:
        labels = a.get("labels", {}) if isinstance(a, dict) else {}
        alert_name = labels.get("alertname") or a.get("ruleId") or str(a)[:100]
        self_healing.handle_alert(alert_name)
        triggered.append(alert_name)
    return {"status": "ok", "triggered": triggered, "state": asdict(self_healing.get_state())}

Caption: Self-healing Ops Console after injecting cost chaos.

My ultimate goal is for the agent to react seamlessly to its own telemetry — catching an unexpected latency spike or cost explosion and downgrading itself to a cheaper, lighter model mid-session without human intervention. Wrestling with webhook payload verification and avoiding cyclic retry loops was challenging, but seeing the event stream sync up proved the architecture worked.

What I'd tell someone setting this up today
Use Foundry: Do not use the old docker-compose path. Check the SigNoz repo's current docs.

Go beyond auto-instrumentation: The HTTP spans tell you that something happened; custom span attributes on your business logic (cost, tokens, model name) are what let you debug why.

Build dashboards early: Creating a panel early surfaces unit and precision issues that you won't notice just staring at raw trace data.

If you're setting up SigNoz for the first time, budget about 10 minutes from zero to your first trace. This project is part of my submission for the Agents of SigNoz Hackathon by WeMakeDevs. Observability-powered self-healing agents are incredibly powerful, and I'm excited to push this setup to its absolute limit.