DEV Community: Akshay

The Loudest Alert Was Wrong: Building RootSpan with SigNoz

Akshay — Sun, 26 Jul 2026 15:53:54 +0000

An alert can be perfectly accurate and still point at the wrong place to begin investigating.

In my hackathon incident, checkout returned errors and the gateway was slow, but neither service was locally broken. Both were waiting on inventory.reserve, where a scoped timeout had started. That gap—between the loudest symptom and the first broken operation—is why I built RootSpan for the Agents of SigNoz hackathon.

RootSpan is a read-only incident-correlation system. It compares matched healthy and failing telemetry cohorts, ranks the first local divergence, shows supporting and contradicting evidence, and hands the decision to a human. It never deploys, rolls back, or restarts anything.

The incident I wanted to solve

I built a three-service Python lab with this request path:

traffic -> gateway.checkout -> checkout.place_order -> inventory.reserve
                                                    -> inventory.db.select

Healthy traffic used inventory-v1 and a control flag. The failing cohort was deliberately narrow: ap-south-1, inventory-v2, and the async-reserve flag. When the resettable fault switch was enabled, inventory.reserve waited 350 ms and returned a timeout. Gateway and checkout then failed too.

That shape matters. A naive “first red span” or “slowest service” rule can blame a parent whose duration merely includes downstream waiting. RootSpan instead asks: what changed repeatedly in failing traces, stayed normal in matched healthy traces, and consumed time locally rather than inheriting it?

SigNoz was the evidence plane, not a screenshot at the end

I installed self-hosted SigNoz and its MCP server through Foundry using the committed casting.yaml and generated lock file:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

SigNoz served six distinct roles in RootSpan:

It stored the lab's OpenTelemetry traces, structured logs, and custom metrics.
A trace-based checkout error-rate rule detected customer-visible impact.
Its Alertmanager-compatible webhook opened an idempotent RootSpan incident.
The SigNoz MCP server supplied bounded trace, log, metric, and query results.
Query Builder v5 produced reproducible latency and blast-radius aggregations.
SigNoz observed RootSpan itself, including correlation stages and Sentinel Mesh spans.

This is possible because SigNoz treats logs, metrics, and traces as connected OpenTelemetry signals, while its Query Builder supports filtering, aggregation, percentiles, grouping, and formulas across those surfaces. OpenTelemetry context propagation preserved the trace relationship across the gateway, checkout, and inventory HTTP calls; trace and span IDs also made the JSON logs directly correlatable.

The application exported all three signals through OTLP. The two custom counters were intentionally boring: rootspan.lab.requests and rootspan.lab.failures. Their job was to answer a decision, not decorate a dashboard.

Querying SigNoz without putting an LLM in the data path

The most important architecture decision was to use the SigNoz MCP server as a normal programmatic client. A model was not required to decide what evidence existed.

For each incident, RootSpan used typed, bounded calls:

signoz_search_traces          -> healthy and failing trace IDs
signoz_get_trace_details      -> complete bounded trace trees
signoz_search_logs            -> timeout fingerprints and trace-linked examples
signoz_execute_builder_query  -> latency comparison and blast-radius groups
signoz_query_metrics          -> metric corroboration

Every call stored its tool name, typed arguments, time range, response hash, duration, status, and SigNoz deep link. The correlation core depended on a TelemetryGateway contract, so replay fixtures and live MCP results returned the same domain types. That kept the algorithm testable when the live stack was offline without creating a second definition of evidence.

Runtime credentials were Viewer-only. Bootstrap credentials could create the alert, dashboard, webhook channel, and service account, but the running investigator could only read. That separation made the human-approval boundary enforceable rather than aspirational.

How first-divergence ranking works

RootSpan aligns the same operations across healthy and failing traces and calculates both inclusive duration and exclusive, or self, duration. For each operation it records:

prevalence in failing versus healthy traces;
local error-rate lift;
inclusive and exclusive duration ratios;
the fraction of the total shift attributable to local work;
independent signal support;
contradiction and cohort-coverage penalties.

The key lesson was that inclusive latency alone is misleading. In the golden incident, the gateway and checkout spans became slow because their child call was slow. Their self-duration stayed near baseline. inventory.reserve showed the large local shift and the timeout, so it ranked first.

The score remains inspectable, and RootSpan calls the result a ranked hypothesis—not proven causality. If either cohort is missing, has fewer than two usable traces, falls below 50% requested coverage, or no operation crosses the evidence threshold, the incident becomes INSUFFICIENT_EVIDENCE. Returning no diagnosis is a feature.

The Sentinel Mesh: parallel observation with deterministic authority

Because the hackathon focused on agents, I wanted more than a single process with an “AI” label. Each live incident creates a logical Sentinel Mesh: gateway, checkout, inventory, and database observers run bounded read-only work concurrently.

One leader is elected through an atomic SQLite lease. Leadership is not decided by a model or a vote. If the leader fails, the lease generation advances and a healthy follower takes over. A failed follower remains visible as degraded coverage; it cannot erase evidence from the others. The deterministic ranker—not the sentinels—owns scoring and abstention.

SigNoz made that workflow observable through spans such as sentinel.leader.elect, sentinel.delegate, sentinel.observe, and sentinel.leader.failover, alongside cohort.select, trace.align, divergence.rank, and brief.compile. The investigation system therefore had to meet the same observability standard as the system it inspected.

The setup problem I did not expect

The hardest integration issue was telemetry ingestion, not ranking. In my local Foundry cast, the OpAMP-managed ingester could receive a pre-onboarding configuration with nop receivers and exporters. Containers appeared healthy while OTLP ports refused connections.

I solved this with a small application-owned OpenTelemetry Collector bridge. It accepted OTLP from RootSpan and the lab, then used SigNoz's native ClickHouse exporters against the same Foundry telemetry store. This was deployment scaffolding, not a second product database. More importantly, I added make telemetry-check assertions for complete three-service traces, both custom metrics, timeout logs, all six correlation-stage span names, and sentinel observation spans. A green container was no longer accepted as proof that telemetry worked.

What the measurements actually say

I evaluated correctness separately from execution speed. The deterministic suite contains 14 labeled simulations, including local failures at four operations, partial prevalence, shuffled input order, missing or undersized cohorts, and normal telemetry that should trigger abstention.

Running make evaluate produced:

100% top-1 localization and 1.0 mean reciprocal rank;
100% abstention recall and 0% false diagnoses;
100% citation integrity, deep-link coverage, and query provenance;
stable ranking across 280 seeded reorder trials;
2.557 ms p95 for the core at 50 healthy plus 50 failing traces on this run.

The final deployed make live-verify gate measured a different boundary: five complete MCP/SigNoz investigations. It ranked inventory.reserve first in 5/5 runs with 1002.8 ms request-to-ready p50 and 1283.3 ms p95. Those numbers describe one controlled local scenario, not production root-cause accuracy or an SLA improvement.

To reproduce the path after provisioning Foundry:

make bootstrap-signoz
make app-up
make live-verify
make evaluate

What I would tell my past self

Match cohorts before writing a clever scorer. Separate local work from inherited latency. Store contradictions with the supporting evidence. Test abstention as seriously as success. And instrument the investigator early—otherwise “the agent is slow” becomes another incident with no trace.

Most importantly, an LLM does not need to own the evidence pipeline for a project to be agentic. RootSpan's useful autonomy is bounded: coordinate observers, gather verified telemetry, and prepare the next human decision. Deterministic code decides what the data supports; a future model can explain that packet, but it cannot manufacture evidence or acquire production authority.

RootSpan did not fix the inventory timeout. It did something I trust more: it turned an upstream checkout alert into a cited, reproducible explanation of the first local divergence—and then stopped at the human handoff.

Project links: source code, architecture and evaluation, and the official SigNoz overview.

Before the Agent Acts: Designing Evidence-First Incident Triage with SigNoz

Akshay — Sun, 19 Jul 2026 18:29:24 +0000

An alert tells me where a system is hurting. It does not always tell me where the failure began.

That difference is expensive during an incident. The loudest signal may come from an API gateway or checkout service, while the first locally abnormal operation sits several calls downstream. Engineers lose time moving between trace search, log search, metric charts, deployment context, and Slack—then reconstructing the same evidence for the next responder.

For the Agents of SigNoz hackathon, I want to build RootSpan: a read-only incident correlation agent that shortens that path. It will not patch code, restart production, or claim certainty it cannot prove. Its job is to find the first meaningful divergence between healthy and failing traffic, measure the blast radius, and give a human an evidence-linked handoff.

Before writing the agent, I built a local SigNoz lab and investigated a real failure manually. That gives the project a test oracle: if the agent cannot reproduce a careful human investigation, it should not recommend an action.

The investigation contract

RootSpan should answer six questions in order:

Which service, operation, and time window are affected?
What do failing traces share that healthy traces do not?
Where does the error first appear with local evidence?
Which logs corroborate or contradict that trace-level hypothesis?
Is this one unusual request or a population-level incident?
What raw SigNoz evidence should the responder open next?
This is investigation, not autonomous remediation. A human remains responsible for deciding whether to roll back, change configuration, contact an owner, or continue observing.

Building the local lab with Foundry

The hackathon requires SigNoz to be installed through Foundry. Foundry represents the deployment with casting.yaml and generates casting.yaml.lock, which makes the environment reproducible.

I used foundryctl v0.2.14 and this local Docker configuration:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz-warmup
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

Then I ran:

foundryctl cast -f casting.yaml --no-updater

The cold image download was the slow part. Once the images were cached, the successful cast completed in about 15 seconds. The stack exposed the SigNoz UI on localhost:8080, OTLP ingestion on 4317 and 4318, and the MCP endpoint on localhost:8000/mcp.

One setup detail was easy to miss: before the first workspace was registered, the collector accepted a placeholder configuration. Registration triggered an OpAMP configuration update that activated the real trace, metric, and log pipelines. If OTLP appears reachable but no data arrives, finish the workspace setup and inspect the collector configuration rather than immediately blaming the application instrumentation.

Sending telemetry worth correlating

For the warm-up I used the official OpenTelemetry Demo Lite, configured for two requests per second. It produces traces, metrics, and logs from 14 services across Go, JavaScript, and Python:

accounting, ad, browser-frontend, cart, checkout, currency, email, fraud-detection, frontend, payment, product-catalog, quote-python, recommendation, and shipping.

The demo also generates controlled failures. The payment service rejects about 5% of calls, while the email service simulates an SMTP failure on about 2%. That is useful for correlation because successful and failing cohorts exist at the same time under the same deployment.

This reinforced a basic observability lesson: an agent cannot recover structure the telemetry never recorded. Stable service.name and operation names, propagated trace context, error status, and trace-aware logs are more valuable than clever prompting.

Following a failure through the trace

I filtered for traces containing errors and selected a representative trace:

d4998f7399786e8dd3405d3dfb2fcd62

It crossed all 14 services and contained 94 spans. The visible transaction lasted about 16 ms, so this was not primarily a latency investigation.

The waterfall showed two red spans around order confirmation. The email service's server span, sendOrderConfirmation, carried the first clear local evidence:

status: Error;
message: SMTP connection failed;
exception event recorded on the email span;
a second email_failed event with smtp_error as the failure reason.
Its checkout ancestor also returned an error, but the email span contained the local exception. That distinction is exactly what RootSpan must preserve: an upstream error is not automatically the origin of the failure.

I would describe this as the first locally evidenced failure, not an absolute root cause. The SMTP dependency itself is not instrumented, so the trace cannot prove why the connection failed.

Correlating the logs without a time-window guessing game

The selected email span did not have a log record attached directly to its span ID. SigNoz still let me pivot from the trace to all logs carrying the same trace ID.

That view showed the sequence clearly:

SendOrderConfirmation
SendOrderConfirmation failed
Email failed
PublishToKafka
Order placed successfully

This is more informative than searching globally for “SMTP”. The error is real, but the order workflow continued. Accounting and fraud-detection consumed the order, and checkout logged Order placed successfully.

That evidence changes the incident classification. The likely customer impact is a missing confirmation email, not a failed order. A responder might contact the email-service owner or inspect the SMTP dependency instead of rolling back the entire checkout service.

There is also a useful contradiction here: the failing trace contains a successful business outcome. RootSpan should include contradictory evidence in its handoff rather than hiding it to make a cleaner story.

Checking population-level impact

One trace is evidence; it is not an incident by itself.

In the 30-minute service view captured below, SigNoz showed all 14 applications with latency, error-rate, and throughput columns. Email had a 2.12% error rate with 0.80 ms p99 latency. Payment showed a 4.71% error rate with 0.74 ms p99 latency. The rates matched the demo's controlled failure behavior closely enough to confirm that the trace was part of a recurring population, not a single corrupt sample.

This is still not a healthy-versus-incident comparison. A production-quality agent must compare explicit cohorts and retain counts as well as percentages. A 5% error rate across 20 requests is not equivalent to 5% across 20 million.

What the agent should automate

The manual warm-up exposed the mechanical work that an MCP-driven agent can remove:

query a bounded set of failing and healthy traces;
group failures by service and operation;
reconstruct the full parent-child trace rather than stopping at the matching span;
rank the first locally abnormal spans;
fetch logs by trace and span context;
check service-level error, latency, and throughput signals;
return direct evidence links, timestamps, query arguments, and confidence reasons.
The agent's output should look less like a chatbot answer and more like an incident handoff:

Hypothesis: email confirmation is degraded after checkout succeeds
First local evidence: email/sendOrderConfirmation
Trace evidence: SMTP exception; checkout ancestor propagated the error
Corroborating logs: SendOrderConfirmation failed; Email failed
Contradictory evidence: order published and placed successfully
Observed impact: recurring email failures; order path continues
Next human checks: SMTP dependency health, email service changes, affected users

Every line should be clickable back to SigNoz or reproducible from a stored query. If the agent cannot show its evidence, it should lower its confidence instead of becoming more persuasive.

Why I am not starting with auto-remediation

Most teams will not allow a new AI agent to edit code or restart production during a live incident—and they should not have to.

The first valuable product is smaller and safer: reduce the time between alert and the first correct human decision. A read-only correlator has a realistic adoption path because responders can verify every claim, learn where the model is weak, and measure whether it actually reduces triage time.

Telemetry correlation is also not causality. A span may wait on an uninstrumented dependency. A deployment may be near an alert but unrelated. A “healthy” baseline may contain a different traffic mix. Human judgment is still required to test the hypothesis and choose a production action.

What I will build during the hackathon

The implementation will be intentionally narrow:

A Python correlation engine that queries SigNoz through MCP and compares healthy and failing trace cohorts.
A deterministic scoring layer for local errors, cohort specificity, corroborating logs, blast radius, and contradictory evidence.
A small React investigation console that presents hypotheses and deep links, not a fake autonomous command center.
A Go incident lab that produces one repeatable multi-service failure for the demo.
An evidence ledger in SQLite so every conclusion records the query, time window, and source IDs that produced it.
The success metric is not “the model fixed production.” It is whether another engineer can reach and verify the right next action faster.

Conclusion

The warm-up trace already demonstrated why correlation matters. A red checkout span suggested a failed purchase. The deeper email exception and trace-linked logs showed a narrower reality: confirmation failed, while the order continued successfully.

That is the kind of distinction I want RootSpan to surface consistently. SigNoz already holds the evidence across traces, logs, and service metrics. The agent's job is to assemble it into a bounded, honest handoff—then let the human decide what to do.