The Loudest Alert Was Wrong: Building RootSpan with SigNoz

An alert can be perfectly accurate and still point at the wrong place to begin investigating.

In my hackathon incident, checkout returned errors and the gateway was slow, but neither service was locally broken. Both were waiting on inventory.reserve, where a scoped timeout had started. That gap—between the loudest symptom and the first broken operation—is why I built RootSpan for the Agents of SigNoz hackathon.

RootSpan is a read-only incident-correlation system. It compares matched healthy and failing telemetry cohorts, ranks the first local divergence, shows supporting and contradicting evidence, and hands the decision to a human. It never deploys, rolls back, or restarts anything.

The incident I wanted to solve

I built a three-service Python lab with this request path:

traffic -> gateway.checkout -> checkout.place_order -> inventory.reserve
                                                    -> inventory.db.select

Healthy traffic used inventory-v1 and a control flag. The failing cohort was deliberately narrow: ap-south-1, inventory-v2, and the async-reserve flag. When the resettable fault switch was enabled, inventory.reserve waited 350 ms and returned a timeout. Gateway and checkout then failed too.

That shape matters. A naive “first red span” or “slowest service” rule can blame a parent whose duration merely includes downstream waiting. RootSpan instead asks: what changed repeatedly in failing traces, stayed normal in matched healthy traces, and consumed time locally rather than inheriting it?

SigNoz was the evidence plane, not a screenshot at the end

I installed self-hosted SigNoz and its MCP server through Foundry using the committed casting.yaml and generated lock file:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

SigNoz served six distinct roles in RootSpan:

1. It stored the lab's OpenTelemetry traces, structured logs, and custom metrics.
2. A trace-based checkout error-rate rule detected customer-visible impact.
3. Its Alertmanager-compatible webhook opened an idempotent RootSpan incident.
4. The SigNoz MCP server supplied bounded trace, log, metric, and query results.
5. Query Builder v5 produced reproducible latency and blast-radius aggregations.
6. SigNoz observed RootSpan itself, including correlation stages and Sentinel Mesh spans.

This is possible because SigNoz treats logs, metrics, and traces as connected OpenTelemetry signals, while its Query Builder supports filtering, aggregation, percentiles, grouping, and formulas across those surfaces. OpenTelemetry context propagation preserved the trace relationship across the gateway, checkout, and inventory HTTP calls; trace and span IDs also made the JSON logs directly correlatable.

The application exported all three signals through OTLP. The two custom counters were intentionally boring: rootspan.lab.requests and rootspan.lab.failures. Their job was to answer a decision, not decorate a dashboard.

Querying SigNoz without putting an LLM in the data path

The most important architecture decision was to use the SigNoz MCP server as a normal programmatic client. A model was not required to decide what evidence existed.

For each incident, RootSpan used typed, bounded calls:

signoz_search_traces          -> healthy and failing trace IDs
signoz_get_trace_details      -> complete bounded trace trees
signoz_search_logs            -> timeout fingerprints and trace-linked examples
signoz_execute_builder_query  -> latency comparison and blast-radius groups
signoz_query_metrics          -> metric corroboration

Every call stored its tool name, typed arguments, time range, response hash, duration, status, and SigNoz deep link. The correlation core depended on a TelemetryGateway contract, so replay fixtures and live MCP results returned the same domain types. That kept the algorithm testable when the live stack was offline without creating a second definition of evidence.

Runtime credentials were Viewer-only. Bootstrap credentials could create the alert, dashboard, webhook channel, and service account, but the running investigator could only read. That separation made the human-approval boundary enforceable rather than aspirational.

How first-divergence ranking works

RootSpan aligns the same operations across healthy and failing traces and calculates both inclusive duration and exclusive, or self, duration. For each operation it records:

• prevalence in failing versus healthy traces;
• local error-rate lift;
• inclusive and exclusive duration ratios;
• the fraction of the total shift attributable to local work;
• independent signal support;
• contradiction and cohort-coverage penalties.

The key lesson was that inclusive latency alone is misleading. In the golden incident, the gateway and checkout spans became slow because their child call was slow. Their self-duration stayed near baseline. inventory.reserve showed the large local shift and the timeout, so it ranked first.

The score remains inspectable, and RootSpan calls the result a ranked hypothesis—not proven causality. If either cohort is missing, has fewer than two usable traces, falls below 50% requested coverage, or no operation crosses the evidence threshold, the incident becomes INSUFFICIENT_EVIDENCE. Returning no diagnosis is a feature.

The Sentinel Mesh: parallel observation with deterministic authority

Because the hackathon focused on agents, I wanted more than a single process with an “AI” label. Each live incident creates a logical Sentinel Mesh: gateway, checkout, inventory, and database observers run bounded read-only work concurrently.

One leader is elected through an atomic SQLite lease. Leadership is not decided by a model or a vote. If the leader fails, the lease generation advances and a healthy follower takes over. A failed follower remains visible as degraded coverage; it cannot erase evidence from the others. The deterministic ranker—not the sentinels—owns scoring and abstention.

SigNoz made that workflow observable through spans such as sentinel.leader.elect, sentinel.delegate, sentinel.observe, and sentinel.leader.failover, alongside cohort.select, trace.align, divergence.rank, and brief.compile. The investigation system therefore had to meet the same observability standard as the system it inspected.

The setup problem I did not expect

The hardest integration issue was telemetry ingestion, not ranking. In my local Foundry cast, the OpAMP-managed ingester could receive a pre-onboarding configuration with nop receivers and exporters. Containers appeared healthy while OTLP ports refused connections.

I solved this with a small application-owned OpenTelemetry Collector bridge. It accepted OTLP from RootSpan and the lab, then used SigNoz's native ClickHouse exporters against the same Foundry telemetry store. This was deployment scaffolding, not a second product database. More importantly, I added make telemetry-check assertions for complete three-service traces, both custom metrics, timeout logs, all six correlation-stage span names, and sentinel observation spans. A green container was no longer accepted as proof that telemetry worked.

What the measurements actually say

I evaluated correctness separately from execution speed. The deterministic suite contains 14 labeled simulations, including local failures at four operations, partial prevalence, shuffled input order, missing or undersized cohorts, and normal telemetry that should trigger abstention.

Running make evaluate produced:

• 100% top-1 localization and 1.0 mean reciprocal rank;
• 100% abstention recall and 0% false diagnoses;
• 100% citation integrity, deep-link coverage, and query provenance;
• stable ranking across 280 seeded reorder trials;
• 2.557 ms p95 for the core at 50 healthy plus 50 failing traces on this run.

The final deployed make live-verify gate measured a different boundary: five complete MCP/SigNoz investigations. It ranked inventory.reserve first in 5/5 runs with 1002.8 ms request-to-ready p50 and 1283.3 ms p95. Those numbers describe one controlled local scenario, not production root-cause accuracy or an SLA improvement.

To reproduce the path after provisioning Foundry:

make bootstrap-signoz
make app-up
make live-verify
make evaluate

What I would tell my past self

Match cohorts before writing a clever scorer. Separate local work from inherited latency. Store contradictions with the supporting evidence. Test abstention as seriously as success. And instrument the investigator early—otherwise “the agent is slow” becomes another incident with no trace.

Most importantly, an LLM does not need to own the evidence pipeline for a project to be agentic. RootSpan's useful autonomy is bounded: coordinate observers, gather verified telemetry, and prepare the next human decision. Deterministic code decides what the data supports; a future model can explain that packet, but it cannot manufacture evidence or acquire production authority.

RootSpan did not fix the inventory timeout. It did something I trust more: it turned an upstream checkout alert into a cited, reproducible explanation of the first local divergence—and then stopped at the human handoff.

Project links: source code, architecture and evaluation, and the official SigNoz overview.