DEV Community: AMAN KUMAR MAURYA

I Made an AI Agent Debug Itself Using SigNoz's MCP Server

AMAN KUMAR MAURYA — Sun, 26 Jul 2026 18:24:24 +0000

I asked a terminal command a question — "Why are check_inventory calls failing? Give concrete numbers." — and got back:

In the last 6 hours on the sre-agent service:

Total spans named execute_tool check_inventory: 11
Error spans: 0 (0%)
Fraction with an error: 0 / 11 (0%) Queried directly using signoz_aggregate_traces via the sre-agent service filter. That wasn't me reading a dashboard. That was a second AI agent, with zero prior knowledge of my system, connecting to SigNoz's MCP server, picking the right query tool on its own, and answering with a real number instead of a guess. This post is about building that, and about the four separate ways it broke before it worked.

The problem I was actually trying to solve
AI agents fail quietly. A tool call times out, an LLM call gets slow, a retry loop triples your token spend — and none of that looks like a crash. It looks like nothing, until a user complains. For the Agents of SigNoz hackathon (Track 01: AI & Agent Observability), I didn't want to just wire an agent up to SigNoz and screenshot a trace. I wanted to answer a sharper question: can the observability data itself become the thing a second agent uses to explain what went wrong?

What I built
Two pieces. First, sre-agent: a tool-calling support bot on FastAPI + Gemini, with three tools — a weather lookup, a docs search, and an inventory check I made deliberately unreliable on purpose:

def check_inventory(sku: str) -> str:
with tracer.start_as_current_span("execute_tool check_inventory") as span:
span.set_attribute("gen_ai.tool.name", "check_inventory")
try:
latency = random.choice([0.05, 0.08, 0.1, 0.1, 2.4]) # occasional slow path
time.sleep(latency)
if random.random() < 0.25:
raise RuntimeError(f"inventory-service timeout for sku={sku}")
...
except Exception as exc:
span.record_exception(exc)
span.set_status(Status(StatusCode.ERROR, str(exc)))
tool_call_errors.add(1, {"tool.name": "check_inventory"})
25% failure rate, occasional 2.4-second stalls. Without a real, ugly failure mode, a "look, it's observable!" demo is just theater.

Second, the SRE Sidekick: a completely separate script that connects to SigNoz's MCP server, lists its available tools at runtime, and lets an LLM decide which ones to call to answer a diagnostic question. No hardcoded queries. It has to actually reason about what to ask SigNoz.

Every request to the first agent produces a real OpenTelemetry trace following the GenAI semantic conventions:

invoke_agent support-agent
chat gemini-flash-lite-latest (LLM call decides to use a tool)
execute_tool check_inventory (the flaky one)
chat gemini-flash-lite-latest (LLM produces the final answer)
with gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and an estimated gen_ai.usage.cost_usd on every LLM span. Custom metrics track tool latency and error counts. Logs are correlated to trace_id automatically via the OTel SDK's logging handler. A dashboard and two alert rules (tool error rate, P95 latency) watch all of it — and I built both of those through the MCP server, not by clicking the SigNoz UI.

Four things that broke, in order

Vercel silently breaks sibling imports Locally, from otel_setup import ... in agent/app.py just worked. Deployed to Vercel, every request 500'd with:

File "/var/task/agent/app.py", line 29, in
from otel_setup import (
ModuleNotFoundError: No module named 'otel_setup'
Vercel's Python runtime imports your entry file directly by path — it never adds that file's own directory to sys.path, which every local script implicitly gets for free. Fix:

HERE = Path(file_).resolve().parent
if str(_HERE) not in sys.path:
sys.path.insert(0, str(_HERE))
Three lines, and it only exists because "works locally" and "works as a serverless function" are different claims.

Telemetry can vanish in production while looking fine in dev OpenTelemetry's BatchSpanProcessor buffers spans and flushes on a background timer — usually every few seconds. On a serverless platform, the process can freeze the instant the HTTP response is sent, before that timer ever fires. Traces would work in uvicorn --reload and just disappear on Vercel, with no error anywhere. I added an explicit flush at the end of every request:

@app.post("/chat")
def chat(req: ChatRequest):
try:
return ChatResponse(reply=run_agent(req.message), session_id=req.session_id)
finally:
flush_telemetry() # force_flush() on the tracer/meter/logger providers

"Admin" in the account name doesn't mean Admin the role I created a SigNoz service account, named it admin, and every MCP call still came back:

SigNoz API error: unexpected status 403: only editors/admins can access this resource
SigNoz's newer RBAC model separates creating a service account from granting it a role — the two are unrelated unless you explicitly attach one. The account name meant nothing to the permission system. Once I attached the signoz-admin role under Settings → Service Accounts, every read and write call started working immediately, with no code changes at all. Worth remembering: read the actual error text before assuming your setup is wrong.

An agent's own tool list can blow its token budget The SRE Sidekick's first real run made five sensible MCP tool calls and then died mid-answer:

openai.RateLimitError: Error code: 429 - Quota exceeded for metric:
generativelanguage.googleapis.com/generate_content_free_tier_input_token_count,
limit: 250000, model: gemini-3.5-flash-lite
The cause wasn't the conversation history — it was that I'd handed the model all 41 of SigNoz's MCP tools, full JSON schema included, on every single turn. Some of those schemas (dashboard and alert authoring, especially) run to hundreds of lines. Re-sending that on every hop of a multi-step reasoning loop adds up fast on a free-tier quota. The fix was also just better agent design: scope the tool list to what the task actually needs.

READ_TOOLS = {
"signoz_search_traces", "signoz_aggregate_traces", "signoz_search_logs",
"signoz_aggregate_logs", "signoz_list_services", "signoz_get_service_top_operations",
"signoz_query_metrics", "signoz_list_metrics", "signoz_list_alerts",
}
tool_specs = [spec(t) for t in tools if t.name in READ_TOOLS]
Eleven tools instead of forty-one. The Sidekick's next run finished cleanly and produced the answer at the top of this post.

What I'd tell past-me
Deploy early, even a rough version. Three of these four bugs only exist in production, and I'd rather hit them with two hours of runway left than zero.
An MCP server's tool list is not free. If you're building an agent against a big tool surface, scope it down before you assume the model is "confused" — it might just be starved of context budget by its own tool definitions.
Permission errors are almost always exactly what they say. 403: not authorized meant not authorized — not a bug in my request.
A deliberately broken code path (my flaky check_inventory) is worth more than a hundred clean ones for actually testing whether your observability setup does anything.
Try it
Repo: https://github.com/23f2001033/agents-of-signoz
Live: https://agents-of-signoz.vercel.app — POST /chat with {"message": "..."}
Ask the Sidekick yourself: python sre_sidekick/sidekick.py "your question here"
If you can't observe your AI agents, you don't own them. Mine, at least, can explain itself.

I Built an AI Tutor That Actually Remembers Me — Using Cognee

AMAN KUMAR MAURYA — Thu, 02 Jul 2026 17:01:52 +0000

The problem that broke me

I'm a Data Science student at IIT Madras. Every week I'm juggling multiple courses — Software Engineering, AI, Software Testing — plus hackathons, assignments, and research. My study sessions are fragmented. I pick up a topic, drop it, come back to it three days later, completely confused again.

I tried every AI chat tool to help. They were all useless in the same way.

Ask ChatGPT about gradient descent today, and it's great. Come back next week and say "remember when I was confused about this?" — it doesn't. It can't. Every session is a blank slate. You end up re-explaining your own confusion to the AI every time. You become the memory system for a tool that's supposed to help you learn.

That's AI amnesia. And it makes AI tutors fundamentally broken.

What Cognee actually is (and why it changes everything)

I'd come across Cognee before — an open-source AI memory platform that builds a knowledge graph from your data instead of just chunking it into a vector store. But I hadn't deeply understood why that matters until I started building.

The difference is this:

With normal RAG (what every AI chatbot uses), your notes get turned into floating vector embeddings. Ask a question, retrieve the most similar chunks, send them to an LLM. It works. But it treats your knowledge as a bag of disconnected facts.

With Cognee, the same notes become a graph — concepts as nodes, relationships as edges. "Backpropagation" connects to "gradient descent" connects to "chain rule" connects to "neural network training". When you ask a question, the search doesn't just find the closest text — it reasons over the graph, following relationships to give you answers that understand how concepts connect.

That structural difference is what makes persistent, evolving memory actually possible.

What I built: MemoryTutor

GitHub: github.com/23f2001033/MemoryTutor
Live demo: memorytutor-8xubbwgxarbmzutwjeger2.streamlit.app

MemoryTutor is an AI study tutor with two layers of memory:

Layer 1 — Course knowledge graph
You upload your notes (PDF or text). Cognee ingests them with cognee.add(), then cognee.cognify() extracts entities and relationships, building a knowledge graph stored on disk. Ask a question and cognee.search(SearchType.GRAPH_COMPLETION) reasons over that graph — not just nearest-neighbour lookup.

Layer 2 — Your personal learning profile
Every question you ask, MemoryTutor extracts the topic (via a quick Gemini call). If you ask a follow-up, click "Still confused", or phrase it with confusion signals ("I don't get it", "can you explain again"), the topic gets flagged as a WeakArea. Nail it and click "Got it" — it becomes a MasteredTopic. Both states are saved to user_memory.json and mirrored into the Cognee graph as short memory statements.

What this enables:

Close the app. Come back tomorrow. MemoryTutor greets you:

"Last session you struggled with gradient descent and chain rule — want to revisit them before we move on?"

It actually knows you. It remembers. The tutor genuinely gets smarter about you the more you use it.

The architecture

Streamlit UI (app.py)
│
├── engine.py → cognee.add / cognify / search (async, persistent event loop)
├── tracker.py → WeakArea / MasteredTopic → user_memory.json + graph mirror
└── viz.py → networkx + matplotlib concept graph (colored by struggle level)
│
Cognee (LiteLLM + Gemini)
knowledge graph stored in .cognee_system/ and .data_storage/

The knowledge graph visualization was a deliberate design choice: you can literally see which concepts you know (green), which you're struggling with (red), and how they connect. It makes the invisible visible.

The hard parts

Async in Streamlit is painful

Cognee's API is fully async. Streamlit's script model is synchronous and re-runs top-to-bottom on every interaction. Bridging them required a persistent background event loop that stays alive across script reruns — not a simple asyncio.run() call.

Cognee's default storage location

By default Cognee stores its databases inside its own package directory. That means every pip install -r requirements.txt would wipe your entire knowledge graph. I overrode the storage paths in config.py to point to the project root — a small fix with a big impact for persistence.

Graph noise

Cognee's raw graph includes internal bookkeeping nodes (chunk nodes, summary nodes, document nodes) that aren't meaningful concepts. The viz layer filters these out by node type so only real course concepts appear in the visualization.

Free Gemini tier limits

gemini-1.5-flash and text-embedding-004 were retired after I started building. The app now defaults to gemini/gemini-flash-lite-latest and gemini/gemini-embedding-001 — both current, both free-tier friendly.

What I learned about AI memory

Building this changed how I think about what's missing in most AI tools.

We obsess over context windows — how much the model can hold in mind at once. But the real gap is persistence — what it retains between conversations. A tutor that can hold 200,000 tokens in context but forgets you the moment you close the tab is a very smart, very forgetful professor.

Cognee's graph model is the right approach here because it doesn't just store — it structures. And structured storage lets you do things raw retrieval can't: traverse relationships, reason over connections, surface context the user didn't know to ask for.

The future of AI tools isn't bigger context windows. It's better memory architecture.

Try it

Live demo: memorytutor-8xubbwgxarbmzutwjeger2.streamlit.app
GitHub: github.com/23f2001033/MemoryTutor
Upload any course notes PDF and start asking questions. Come back the next day and see what it remembers.

Built in one day for The Hangover Hackathon by WeMakeDevs × Cognee.

Aman Kumar Maurya — IIT Madras BS Data Science student
The problem that broke me