Built during the Agents of SigNoz hackathon. Source + reproducible Foundry config:
github.com/MTC-123/Taraol
We ran a five-agent research pipeline — planner → researcher → writer → critic → router — each
agent in its own container, talking over HTTP. From the outside it looked healthy: every service
was up, every request returned 200, every model call produced valid text.
But on one run, the writer and critic were caught in a loop across process boundaries — the
writer produced a draft, the critic sent it back, the writer regenerated, forever. Each hop was a
real Gemini call in a different container. And here's the problem that makes distributed agents
genuinely hard:
No single service could see the loop. The planner had already returned. The writer only saw
"another request." The critic only saw "another draft." The loop existed only in the
telemetry — in the relationships between the services.
That is exactly where SigNoz comes in. We built Taraol to
instrument agent systems with OpenTelemetry, compare workflow variants, and — crucially — detect
and contain failures like this from the telemetry itself. SigNoz is the backend: traces, the
Service Map, logs, dashboards, and alerts. We built no custom UI.
This post walks the full loop we built: instrument → experiment → distributed trace → detect →
alert → enforce → audit.
Why distributed agent observability is hard
A single LLM call is easy. A mesh of agents owned by different services is not. Traditional
monitoring answers infrastructure questions ("is the writer up?"); LLM evaluation answers quality
questions ("is the answer good?"). Neither answers the operational ones:
- Which agent hop is making the workflow slow or expensive?
- Are two services stuck calling each other in a loop?
- Is a prompt injection propagating from one agent to the next?
- Can the system react automatically before the loop becomes an outage?
Taraol answers these on open standards — W3C trace context and the OpenTelemetry gen_ai
semantic conventions — so it works with any Python agent and any OTel backend.
Architecture
planner ─HTTP▶ researcher ─HTTP▶ writer ─HTTP▶ critic ─HTTP▶ router
│ │ │ │ │
└──────── @agent / @chat / @tool + W3C traceparent on every hop ───────┘
│
OpenTelemetry / OTLP (gen_ai semconv)
│
SigNoz
┌─────────────────┴──────────────────┐
Traces · Service Map · Logs Query engine
Dashboards · Alerts │
▼
Watcher ──▶ Controller
(reads telemetry, (pause agent /
emits signals) break edge / audit)
Every layer is standard OpenTelemetry. Taraol adds the AI-specific spans, the experiment layer,
and the detection/enforcement services; SigNoz provides tracing, topology, storage, dashboards,
and alerting.
📸 Figure 1 (hero) — the Service Map. (Repo:
docs/service-map.png.) The five agent
services, drawn automatically by SigNoz from propagated trace context.
Caption: "SigNoz reconstructs the agent topology from traces alone — no separate topology
database, no hand-drawn diagram."
Instrumentation: two decorators, and traceparent on every hop
Each agent runs as its own FastAPI service. The LLM and tool steps use Taraol's decorators; the
per-request root uses a context manager (it needs the conversation id from the incoming payload):
from taraol import chat, tool, instrument
kit = instrument(service_name) # one-line OpenTelemetry setup per service
@chat(MODEL) # tokens + cost read off the returned response
def complete(prompt: str):
return llm.complete(prompt, MODEL)
@tool # the web search step
def search_sources(query: str) -> str:
return search.hits_to_json(search.web_search(query))
The distributed part is standard trace propagation. Taraol's A2A client injects traceparent
(and baggage) into every outgoing agent-to-agent request, and the receiving service extracts it —
so a request that starts in the planner and ends in the router is one trace in SigNoz. That is
what lets SigNoz draw the Service Map in Figure 1.
Content capture is off by default; we enabled it for the demo so we could read each hop's
prompt and completion under standard gen_ai.* keys.
📸 Figure 2 — a distributed trace. (Repo:
docs/mesh-trace.png.) The flame graph +
waterfall across all five services, with achatspan selected so the right panel shows
gen_ai.input.messages,gen_ai.usage.output_tokens, andagentmesh.cost.direct_usd.
Caption: "One trace spanning five containers. Each hop carries the previous agent's output
as this agent's input — genuine A→B→C data flow, with tokens and cost per hop."
AgentLab: comparing a converging vs a runaway workflow
One healthy run doesn't prove robustness. AgentLab fires the same mesh workload as tagged
variants and compares them on operational telemetry. Here, one stack, two variants — a
baseline that converges and a runaway that storms the writer↔critic loop — under a single
run id, no redeploy (the loop mode rides the request payload):
from taraol import Experiment
(Experiment("converge-vs-runaway", author="Fraol")
.compare("baseline", loop_mode="off") # converges in a few hops
.compare("runaway", loop_mode="storm") # writer ↔ critic loop
.run(fire)) # fire(variant) POSTs to the planner
Every span — in every service — carries experiment.id / experiment.variant /
experiment.run_id, because the tags propagate across the A2A hops too. We compare them in a
SigNoz dashboard (shipped with Taraol as JSON, built with the Query Builder) and via a CLI
that reads the same telemetry back. One real run:
variant cost$ tokens agents avg ms loops breakers fails health
baseline 0.0079 2799 5 50340 0 0 0 89.9
runaway 0.0406 14342 5 130017 1 1 0 59.0
The runaway variant cost ~5× the tokens and ran ~2.5× longer — and the loops and
breakers columns are the important part: they were populated by detection running from
telemetry, not by any single agent noticing.
📸 Figure 3 — the AgentLab dashboard. (Repo:
docs/agentlab-dashboard.pngshows the
concept; for the mesh, capture the dashboard on theconverge-vs-runawayrun.) Panels split by
experiment.variant: direct LLM cost, output tokens, P95 latency, loop count, breaker trips.
Caption: "Same workload, two designs, compared on real operational telemetry — SigNoz is
the dashboard."
The dashboard tells us which variant is abnormal. The trace tells us why.
The distributed trace: the loop nobody could see
Opening Trace Explorer and filtering to the runaway run, the longest trace showed one pair of
operations repeating across services:
writer → critic
writer → critic
writer → critic
...
That single pattern explained all three dashboard symptoms — latency, tokens, and cost all rose
because every revision was another cross-service model call. This was our key lesson:
A dashboard tells you which run is abnormal. A distributed trace tells you why — and
in a mesh, the "why" lives in the edges between services, which is precisely what SigNoz makes
visible.
From detection to enforcement — the part a single agent can't do
Because the loop spans services, no in-process breaker can see it. So Taraol runs a Watcher: a
separate service that queries recent SigNoz telemetry, groups agent-to-agent spans by trace and
edge, looks for repeated traversal with no progress, and emits a structured loop_detected log
record — trace-correlated, and tagged with the experiment variant that caused it. On the run
above it emitted both loop_detected and edge_unhealthy for the writer → critic edge.
The Watcher does not pause anything itself. We kept the operational policy in SigNoz:
loop_detected log
└─▶ SigNoz log alert
└─▶ Controller webhook
└─▶ pause agent / break edge
└─▶ agent_paused / edge_broken ──▶ back to SigNoz (audit)
Why the separation:
- the Watcher decides the telemetry is suspicious;
- SigNoz decides whether that should trigger a response (a configurable alert rule);
- the Controller performs the action (pause the agent, or trip the per-edge breaker so the bad hop short-circuits while the rest of the mesh keeps running);
- the resulting
agent_paused/edge_brokenevent is exported back to SigNoz, so the intervention is itself observable.
Nothing happens silently. An operator gets a visible query, an alert rule, a notification path,
and a full audit trail.
📸 Figure 4 — Logs Explorer filtered to
body = 'loop_detected', showing the signal with
itsexperiment_variantand thewriter → criticedge. Caption: "Detection lives in the
telemetry, not the agent code — the loop no single service could see, caught and made
searchable next to the traces that caused it."📸 Figure 5 (optional) — the audit event. Logs filtered to
body = 'agent_paused'or
'edge_broken', showing the Controller's action correlated back to the trace.
How SigNoz is deployed: reproducible with Foundry
A judge — or any new user — has to reproduce the exact backend. Taraol ships its SigNoz stack as a
Foundry deployment, not a hand-written Compose file:
casting.yaml what you want (SigNoz version, compose flavor)
│ foundryctl cast
▼
casting.yaml.lock every resolved decision pinned (all components + configs)
▼
pours/deployment/ the rendered stack: compose + ClickHouse / keeper / ingester configs
Both files are committed, so the environment reproduces bit-for-bit:
# reproduce the exact backend from the committed spec
foundryctl cast -f demos/research_mesh/signoz/casting.yaml
# or let Taraol boot the bundled deployment in one command
taraol signoz up # → SigNoz UI on :8080, OTLP on :4317
The mesh's compose.yaml includes this Foundry-rendered SigNoz, so the demo's backend is the
Foundry deploy. The same output is also bundled inside the pip package, so pip install taraol && gives any user a Foundry-provisioned SigNoz with no manual Docker steps. To use
taraol signoz up
SigNoz Cloud instead, point OTEL_EXPORTER_OTLP_ENDPOINT at your ingest endpoint — no code
changes.
📸 Figure 6 (optional) — the running stack / Dashboards list with the imported
"AgentLab — Experiment Comparison" dashboard. Caption: "The whole backend comes up with
foundryctl castortaraol signoz up."
What worked, and what we'd change
Held up well:
-
Experiment attributes across process boundaries. Because
experiment.id/variant/run_idpropagate over the A2A hops, every service's spans stay comparable — no mixed results. - Standard trace propagation was enough. W3C context let SigNoz reconstruct the whole cross-service workflow and derive the Service Map — no custom tracing backend.
- Alerts as a policy boundary. Detection and enforcement stayed separate, visible, and configurable.
Honest limitations (a hackathon system, not a finished safety platform):
- pause/breaker state is in memory, so a restart clears it;
- the injection detector is regex-based — a signal, not a complete security layer;
- SigNoz notification-channel setup is still partly manual;
- content capture needs stricter redaction before production;
- the Watcher → alert → Controller path needs stronger end-to-end testing.
We also ship a smaller single-process demo (a four-agent trip planner) for the quickstart path,
but the mesh is where the distributed story — cross-process traces, the Service Map, and
telemetry-driven loop detection — actually lands.
Conclusion
The useful result wasn't another AI dashboard. It was a closed, observable path for a
distributed agent system:
instrument → experiment → distributed trace → detect → alert → enforce → audit
The dashboard showed which variant was unhealthy. The distributed trace showed why — a
writer↔critic loop across containers. The Watcher caught it from telemetry that no single service
could see, and the SigNoz alert path turned that detection into a bounded, audited response.
Every step — spans, experiments, detection signals, and enforcement actions — lives in SigNoz.
Source and the reproducible Foundry configuration are in the
Taraol repository.
Appendix: run it yourself
pip install "taraol[all]"
# reproducible SigNoz backend (Foundry)
docker compose -f demos/research_mesh/compose.yaml up -d --build
# fire the converging-vs-runaway experiment across the five services
docker exec research-mesh-planner-1 python -m research_mesh.experiment
# then open SigNoz on :8080 → Service Map, Traces, the AgentLab dashboard, and
# Logs (body = 'loop_detected')
Top comments (0)