DEV Community

Bhanu Nagpure
Bhanu Nagpure

Posted on

Your agent looks healthy. It isn’t. Building recursive observability with SigNoz MCP

Hackathon: Agents of SigNoz · Track 1 (AI & Agent Observability)

Code: github.com/bhanu-dev82/agentscope

Demo video (~2:43): Demo_AgentScope.mp4

I spent a solid chunk of this week staring at a dashboard that said everything was fine while the agent underneath was spinning in a tool loop. Latency looked normal. Error rate was zero. HTTP kept returning 200. Tokens kept leaving the account.

That gap — “green APM, broken agent” — is what pushed me to build AgentScope for the Agents of SigNoz hackathon.

AgentScope overview — hero and live stats

Our operator console. The tagline is not marketing fluff; it is the bug report we kept hitting.

Why standard APM lies to you about agents

I am not saying traces are useless. They are necessary. But most of the agent failures I care about are semantic:

  • Same tool fires over and over in one conversation (loop)
  • A researcher step burns thousands of tokens on padding or retries
  • The model returns confident garbage and the HTTP layer still celebrates

None of that shows up as a 5xx. Your SLO board stays calm. You only notice when the bill arrives or a user says the answer is nonsense.

For Track 1 I wanted something sharper than “we exported OpenTelemetry to SigNoz.” The question I wrote on a sticky note was:

If an agent can use SigNoz through MCP, can that same agent show up as spans inside SigNoz?

That loop — monitor using the product, then appear inside the product — is what I ended up calling recursive observability. Slightly dramatic name. Accurate description.

AgentScope in one picture

Three moving parts:

  1. A small multi-agent pipeline (Planner → Researcher → Summarizer) on LangGraph + Gemini
  2. Self-hosted SigNoz from Foundry, with MCP turned on
  3. A Watchdog process that polls SigNoz MCP, runs detectors, logs interventions, and writes its own spans back

Architecture strip and chaos sessions in the UI

UI → pipeline → SigNoz → Watchdog. Recent sessions show loop / cost / silent inject modes we actually ran.

Rough data path:

Browser UI (:3000)
    → FastAPI (:8888) runs LangGraph
        → OTLP gRPC (:4317) into SigNoz
            ← Watchdog polls MCP (:8000)
            → Watchdog also exports spans on OTLP
Enter fullscreen mode Exit fullscreen mode

Chaos buttons in the UI do not flip a boolean for the detector. They make the pipeline emit real span shapes. The Watchdog only sees what ClickHouse has after ingestion. That constraint kept me honest.

How I actually used SigNoz

This is the section judges care about, so I will be concrete.

Foundry, not a mystery compose file

Rules want casting.yaml and casting.yaml.lock in the repo. Fair. I cast SigNoz with MCP enabled:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    mode: docker
    flavor: compose
  mcp:
    spec:
      enabled: true
Enter fullscreen mode Exit fullscreen mode 
foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

After that: UI on 8080, OTLP on 4317, MCP on 8000.

I lost half an evening when OTLP stopped accepting data while the UI still looked healthy. Turns out OpAMP had rewritten collector pipelines toward nop receivers. Fix was boring and real: force file-based collector config, recreate the ingester, watch logs until gRPC 4317 came back. If your demo has “no traces,” check the collector before you rewrite application code. I rewrote application code first. Do not be me.

Traces with agent-shaped names

I bootstrapped OpenTelemetry before LangChain imports. Order matters; I learned that the hard way when half the spans vanished.

What landed in SigNoz for service agentscope:

  • invoke_workflow AgentPipeline
  • invoke_agent Planner / Researcher / Summarizer
  • execute_tool web_search
  • generate / LLM spans from the model path
  • attributes like gen_ai.agent.name, token usage, conversation id

I did not invent a new observability religion here. I just refused to ship a single fat “HTTP POST /run” span and call it agent ops.

MCP as a polling data plane (not a chat stunt)

A lot of demos open MCP, ask one question, screenshot the answer, done. Fine for a keynote. Useless as an ops loop.

My Watchdog:

  1. Opens MCP session to http://localhost:8000/mcp
  2. Sends SIGNOZ-API-KEY (created in SigNoz → Settings → API Keys)
  3. Every ~10 seconds calls tools like signoz_aggregate_traces
  4. Parses the response
  5. Runs detectors
  6. Dedupes on (failure_type, trace_id) so it does not spam forever
  7. Writes a critical log (optional webhook if you set one)
  8. Emits invoke_agent Watchdog plus child tool spans such as execute_tool signoz_loop_check

Watchdog intervention log after real detections

Loop, cost spike, silent failure — each with action logged_critical and a real trace id.

One parser bug cost me more time than the detectors: MCP aggregate payloads are columnar (columns + data), not a list of objects. My first code assumed rows-of-dicts, scanned “three rows,” found nothing, and I almost blamed ClickHouse. Always print the raw shape once.

What the detectors look for

Inject Pipeline does Watchdog looks for
Loop Ten web_search tool spans in one trace count of execute_tool * per (trace, name) > 5
Cost spike Heavy token padding on the researcher High input token sum on the trace
Silent failure Garbage tool output + span semantic_failure silent_output Aggregate on name containing semantic_failure

For loops I group by span name (execute_tool web_search) more than nested attributes. Attributes were flaky under time pressure; names were stable in the explorer.

I also had to exclude the Watchdog’s own signoz_*_check tools from the loop rule. Otherwise the monitor alerts on itself and you invent a very philosophical outage.

Metrics, logs, dashboards

Traces carried the story, but I still exported metrics (token usage, durations), bridged Python logs into OTel, and scripted a few dashboards/views through MCP. Not every panel is glamorous. Having API keys + MCP + Foundry in one product is what made the Watchdog feel like a real consumer of SigNoz instead of a side script scraping HTML.

The live path I show in the video

Live Demo — chaos inject while HTTP stays green

Inject loop / cost / silent. Status can still be Complete with a 200 path.

Hero path:

  1. Open Live Demo, hit Inject loop
  2. Pipeline finishes. No HTTP error theater. Error rate still zero.
  3. Wait. I mean it — 15 to 40 seconds for OTLP batch + ClickHouse. The UI even says so. Instant detection would be a lie.
  4. Watchdog tab lights up: tool web_search called 10 times, threshold 5, logged_critical, real trace_id
  5. Jump to SigNoz Traces

SigNoz Traces — invoke_agent Watchdog and signoz_*_check tools

This is the recursive proof. Same service name. Monitor spans next to pipeline traffic.

When invoke_agent Watchdog and execute_tool signoz_loop_check showed up next to the pipeline, the sticky note question was answered. The monitor is not invisible. It is another agent-shaped workload on the same backend.

Full walkthrough is in the demo video (~2:43). Deck is in the repo if you want slides.

Things that bit me (so you skip them)

Ingestion lag. I almost built a fake “instant alert” path for the demo. Glad I did not. Real ClickHouse lag is part of the product story.

Self-alerting. Exclude your own MCP tool spans from loop detection. Sounds obvious after midnight. Less obvious before.

Collector config vs UI health. SigNoz UI up ≠ OTLP path healthy. Verify 4317 when traces disappear.

Attribute worship. Nice when they index cleanly. For hackathon detectors, span names got me further.

English + live demo. I am not a native English speaker. I wrote a short script, practiced once off camera, and recorded with OBS. Slow is fine. Clear inject → detect → SigNoz is better than fast mumbling.

Stack, short

  • SigNoz via Foundry (casting.yaml, MCP, API keys, traces/metrics/logs)
  • Python FastAPI + LangGraph + Gemini
  • OTel traces, metrics, logs to OTLP
  • React operator UI for inject + interventions
  • Repo: bhanu-dev82/agentscope

Run sketch (details in README):

git clone https://github.com/bhanu-dev82/agentscope.git
cd agentscope
foundryctl cast -f casting.yaml
cp .env.example .env   # SIGNOZ_API_KEY, GEMINI_API_KEY
python3 -m venv venv && source venv/bin/activate
pip install -r requirements.txt && python -m src.run
cd ui && npm i && npm run dev -- --port 3000
Enter fullscreen mode Exit fullscreen mode

Closing

I did not invent agent monitoring. I wired a small multi-agent app to SigNoz the way the hackathon asks you to — Foundry, deep signals, MCP as something an agent actually calls on a timer — and I made the monitor leave fingerprints in the same traces UI.

If your agents only look healthy, dig one layer under HTTP. And if you build a watchdog, put it on the board too.

Recursive observability: observe agents with an agent, through SigNoz MCP. The monitor is itself monitored, in the same product.

Links: GitHub · Video · Deck · SigNoz

Top comments (0)