I asked a terminal command a question — "Why are check_inventory calls failing? Give concrete numbers." — and got back:
In the last 6 hours on the sre-agent service:
- Total spans named execute_tool check_inventory: 11
- Error spans: 0 (0%)
- Fraction with an error: 0 / 11 (0%) Queried directly using signoz_aggregate_traces via the sre-agent service filter. That wasn't me reading a dashboard. That was a second AI agent, with zero prior knowledge of my system, connecting to SigNoz's MCP server, picking the right query tool on its own, and answering with a real number instead of a guess. This post is about building that, and about the four separate ways it broke before it worked.
The problem I was actually trying to solve
AI agents fail quietly. A tool call times out, an LLM call gets slow, a retry loop triples your token spend — and none of that looks like a crash. It looks like nothing, until a user complains. For the Agents of SigNoz hackathon (Track 01: AI & Agent Observability), I didn't want to just wire an agent up to SigNoz and screenshot a trace. I wanted to answer a sharper question: can the observability data itself become the thing a second agent uses to explain what went wrong?
What I built
Two pieces. First, sre-agent: a tool-calling support bot on FastAPI + Gemini, with three tools — a weather lookup, a docs search, and an inventory check I made deliberately unreliable on purpose:
def check_inventory(sku: str) -> str:
with tracer.start_as_current_span("execute_tool check_inventory") as span:
span.set_attribute("gen_ai.tool.name", "check_inventory")
try:
latency = random.choice([0.05, 0.08, 0.1, 0.1, 2.4]) # occasional slow path
time.sleep(latency)
if random.random() < 0.25:
raise RuntimeError(f"inventory-service timeout for sku={sku}")
...
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, str(exc)))
tool_call_errors.add(1, {"tool.name": "check_inventory"})
25% failure rate, occasional 2.4-second stalls. Without a real, ugly failure mode, a "look, it's observable!" demo is just theater.
Second, the SRE Sidekick: a completely separate script that connects to SigNoz's MCP server, lists its available tools at runtime, and lets an LLM decide which ones to call to answer a diagnostic question. No hardcoded queries. It has to actually reason about what to ask SigNoz.
Every request to the first agent produces a real OpenTelemetry trace following the GenAI semantic conventions:
invoke_agent support-agent
chat gemini-flash-lite-latest (LLM call decides to use a tool)
execute_tool check_inventory (the flaky one)
chat gemini-flash-lite-latest (LLM produces the final answer)
with gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and an estimated gen_ai.usage.cost_usd on every LLM span. Custom metrics track tool latency and error counts. Logs are correlated to trace_id automatically via the OTel SDK's logging handler. A dashboard and two alert rules (tool error rate, P95 latency) watch all of it — and I built both of those through the MCP server, not by clicking the SigNoz UI.
Four things that broke, in order
- Vercel silently breaks sibling imports Locally, from otel_setup import ... in agent/app.py just worked. Deployed to Vercel, every request 500'd with:
File "/var/task/agent/app.py", line 29, in
from otel_setup import (
ModuleNotFoundError: No module named 'otel_setup'
Vercel's Python runtime imports your entry file directly by path — it never adds that file's own directory to sys.path, which every local script implicitly gets for free. Fix:
HERE = Path(file_).resolve().parent
if str(_HERE) not in sys.path:
sys.path.insert(0, str(_HERE))
Three lines, and it only exists because "works locally" and "works as a serverless function" are different claims.
- Telemetry can vanish in production while looking fine in dev OpenTelemetry's BatchSpanProcessor buffers spans and flushes on a background timer — usually every few seconds. On a serverless platform, the process can freeze the instant the HTTP response is sent, before that timer ever fires. Traces would work in uvicorn --reload and just disappear on Vercel, with no error anywhere. I added an explicit flush at the end of every request:
@app.post("/chat")
def chat(req: ChatRequest):
try:
return ChatResponse(reply=run_agent(req.message), session_id=req.session_id)
finally:
flush_telemetry() # force_flush() on the tracer/meter/logger providers
- "Admin" in the account name doesn't mean Admin the role I created a SigNoz service account, named it admin, and every MCP call still came back:
SigNoz API error: unexpected status 403: only editors/admins can access this resource
SigNoz's newer RBAC model separates creating a service account from granting it a role — the two are unrelated unless you explicitly attach one. The account name meant nothing to the permission system. Once I attached the signoz-admin role under Settings → Service Accounts, every read and write call started working immediately, with no code changes at all. Worth remembering: read the actual error text before assuming your setup is wrong.
- An agent's own tool list can blow its token budget The SRE Sidekick's first real run made five sensible MCP tool calls and then died mid-answer:
openai.RateLimitError: Error code: 429 - Quota exceeded for metric:
generativelanguage.googleapis.com/generate_content_free_tier_input_token_count,
limit: 250000, model: gemini-3.5-flash-lite
The cause wasn't the conversation history — it was that I'd handed the model all 41 of SigNoz's MCP tools, full JSON schema included, on every single turn. Some of those schemas (dashboard and alert authoring, especially) run to hundreds of lines. Re-sending that on every hop of a multi-step reasoning loop adds up fast on a free-tier quota. The fix was also just better agent design: scope the tool list to what the task actually needs.
READ_TOOLS = {
"signoz_search_traces", "signoz_aggregate_traces", "signoz_search_logs",
"signoz_aggregate_logs", "signoz_list_services", "signoz_get_service_top_operations",
"signoz_query_metrics", "signoz_list_metrics", "signoz_list_alerts",
}
tool_specs = [spec(t) for t in tools if t.name in READ_TOOLS]
Eleven tools instead of forty-one. The Sidekick's next run finished cleanly and produced the answer at the top of this post.
What I'd tell past-me
Deploy early, even a rough version. Three of these four bugs only exist in production, and I'd rather hit them with two hours of runway left than zero.
An MCP server's tool list is not free. If you're building an agent against a big tool surface, scope it down before you assume the model is "confused" — it might just be starved of context budget by its own tool definitions.
Permission errors are almost always exactly what they say. 403: not authorized meant not authorized — not a bug in my request.
A deliberately broken code path (my flaky check_inventory) is worth more than a hundred clean ones for actually testing whether your observability setup does anything.
Try it
Repo: https://github.com/23f2001033/agents-of-signoz
Live: https://agents-of-signoz.vercel.app — POST /chat with {"message": "..."}
Ask the Sidekick yourself: python sre_sidekick/sidekick.py "your question here"
If you can't observe your AI agents, you don't own them. Mine, at least, can explain itself.
Top comments (0)