DEV Community: Tyson

From a Live Voice Prompt to a Safe Canvas Draft: Observing Sidechalk End to End with SigNoZ

Tyson — Mon, 27 Jul 2026 02:35:03 +0000

Project: Sidechalk on GitHub

Hackathon: WeMakeDevs Agents of SigNoZ

Track: Track 01 — AI & Agent Observability

Video: Watch the Sidechalk x SigNoZ demo

In my final live demo, I asked Sidechalk to inspect an inclined-plane diagram, reason about its missing force, and add one yellow friction arrow as a draft. The application saw the authorized canvas, kept the first two answers conversational, and created a visible proposal only after the third instruction. The learner could inspect, accept, or reject it; an accepted change could still be undone.

The SigNoZ command center captured the same run. It showed token series for both gpt-realtime-2.1 and gpt-5.6-terra, structured-provider duration, complete agent-turn duration, estimated model cost, one fallback event, and background-worker activity. A second provisioned dashboard covers proposal validation, decisions, safety blocks, and authoritative realtime delivery.

Getting there exposed a real blind spot. In an earlier run, the learner-facing workflow succeeded while most AI panels stayed empty and only worker telemetry appeared. The problem was not SigNoZ or metric-export delay: I had instrumented the typed assistant route, but the live WebRTC sideband followed a different lifecycle.

Fixing that gap forced me to answer a harder question than “Did the model respond?”: Can I observe the complete agent path that the learner actually used?

What Sidechalk is trying to make safe

Sidechalk is a collaborative living notebook. A learner can draw on an Excalidraw canvas, work in a document, import media, and speak or type to an assistant. The assistant can inspect only the authorized Sidechalk page and propose arrows, handwriting, circles, movement, comments, citations, replacement, or recoverable deletion.

Its central invariant is draft first. Model output cannot directly mutate the canonical Yjs document. AI marks stay visually distinct until the learner explicitly accepts them, and accepted work has a deterministic inverse for undo.

For the demo I use an incomplete free-body diagram: a block on an incline with mg, N, and a velocity arrow down the slope, but no friction force. The conversation is deliberately split into three turns:

“What can you see on the board? Do not change anything yet.”
“The velocity arrow shows the block moving down the slope. In which direction should friction act? Do not draw yet.”
“Add one yellow arrow up the incline and write friction beside it. Change nothing else, and keep it as a draft.”

The first two turns test fresh visual context, conversation memory, and physics reasoning without authorizing a surface operation. Only the third turn authorizes a draft.

I arrived at this scenario after another real limitation: an imported PNG is one flattened canvas element. Asking the agent to delete one arrow baked into that image is not the same as deleting an Excalidraw vector element. Instead of hiding that mismatch, I changed the task to a useful additive correction that the surface contract can perform honestly.

Before the correction: the diagram shows the block moving down the slope, but the friction force is missing.

After the voice instruction: Sidechalk draws friction up the incline as a private proposal; the learner still controls accept or reject.

One editing turn crosses more than one model system

The system is a TypeScript monorepo with Next.js and Excalidraw on the web, Fastify for authoritative APIs, Supabase for authentication and durable state, Yjs/Hocuspocus for collaboration, Redis/BullMQ for distributed work, and a constrained Python STEM verifier. OpenAI handles live voice and multimodal proposals; provider adapters also exist for Google and Anthropic.

Next.js / Excalidraw
        |
        v
Fastify API ----> authorized surface context
        |-------> OpenAI Realtime voice response
        |-------> structured proposal model
        |-------> verifier / citation checks
        |-------> proposal validation and persistence
        |-------> explicit accept, reject, or undo
        v
Yjs + Hocuspocus ----> collaborators

Instrumented services --OTLP/HTTP--> self-hosted SigNoZ
SigNoZ MCP --read only-----> Sidechalk Incident Copilot

The Node API, realtime, and worker services export traces, metrics, and selected logs. The Python verifier exports traces, while the Next.js service uses @vercel/otel. Automatic instrumentation captured HTTP, fetch, and supported library activity, but it could not name the product boundaries I cared about. I added manual spans at each trust boundary:

Boundary	Spans	What it answers
Learner turn	`sidechalk.ai.turn`	Did the complete typed or voice/proposal turn finish?
Authorized context	`sidechalk.ai.context.build`	Did the model receive the current permitted surface?
Provider	`sidechalk.gen_ai.invoke`	Which model call was slow, cancelled, or failed?
Verification	`sidechalk.verifier.check`, `sidechalk.citation.verify`	Did deterministic or source checks run?
Draft safety	`sidechalk.proposal.validate`, `sidechalk.proposal.persist`	Was the model output valid and stored only as a draft?
Human control	`sidechalk.proposal.decide`, `sidechalk.proposal.undo`	What happened after the learner explicitly decided?
Collaboration	`sidechalk.realtime.authoritative_update`	Was the accepted binary update published?
Background work	`sidechalk.worker.job`	Did asynchronous processing continue outside the request?

That manual instrumentation follows SigNoZ's guidance for tracing business operations that framework instrumentation cannot explain. It also gives each failure a useful boundary: provider, validator, persistence, human decision, or realtime publication.

The blind spot: typed turns and live voice were different systems

My typed /turns route already wrapped the complete workflow and recorded custom metrics. This shortened excerpt keeps the real span and metric names while omitting unrelated validation and streaming code:

return withSpan("sidechalk.ai.turn", {
  "sidechalk.channel": "typed",
  "sidechalk.ai.mode": mode,
}, async () => {
  const canonical = await withSpan(
    "sidechalk.ai.context.build",
    { "sidechalk.surface.type": page.type },
    () => productionRepository.prepareTurn(request, conversationId, userId),
  );

  const response = await withSpan(
    "sidechalk.gen_ai.invoke",
    { "gen_ai.operation.name": "chat" },
    () => createProviderResponse(canonical, { adapters }),
  );

  await withSpan("sidechalk.proposal.validate", {}, validateProposal);
  await withSpan("sidechalk.proposal.persist", {}, () =>
    productionRepository.persistAssistant(response, userId, reservationId),
  );
});

Live voice does not remain inside one HTTP request. A browser sends audio through WebRTC, a server-side sideband receives provider events, the browser captures the current authorized page, and tool calls may pause while a draft is rendered and acknowledged. The original implementation persisted those events correctly, but never called recordAiTurn() and never created the equivalent root/provider spans.

The pre-fix dashboard screenshot made the gap obvious:

Before the fix: successful live-voice activity was absent from the AI panels because the Realtime sideband had not yet been instrumented.

Waiting longer could never repair an uninstrumented path.

Fixing an asynchronous Realtime trace

The fix was not to add a timer around the endpoint that triggers response.create; that request ends before the asynchronous provider lifecycle does. I had to keep a turn span alive across sideband events and create a child provider span for each Realtime response. The following excerpt is shortened, but the lifecycle and attributes are taken from the running implementation:

if (!session.activeTurn) {
  session.activeTurn = {
    startedAtMs: performance.now(),
    span: realtimeTracer.startSpan("sidechalk.ai.turn", {
      attributes: safeAttributes({
        "sidechalk.channel": "voice",
        "sidechalk.ai.mode": session.mode,
        "sidechalk.surface.type": context.board.page.type,
        "sidechalk.visual.available": context.visualAvailable,
      }),
    }),
    inputTokens: 0,
    outputTokens: 0,
    responseCount: 0,
  };
}

const parent = trace.setSpan(otelContext.active(), session.activeTurn.span);
session.activeResponse = {
  startedAtMs: performance.now(),
  span: realtimeTracer.startSpan("sidechalk.gen_ai.invoke", {
    attributes: safeAttributes({
      "gen_ai.operation.name": "realtime",
      "gen_ai.provider.name": "openai",
      "gen_ai.request.model":this.options.environment.OPENAI_REALTIME_MODEL,
    }),
  }, parent),
};

When response.done arrives, Sidechalk extracts only numeric usage metadata, ends the provider span, and either continues through a tool call or closes the learner turn:

const calls = completedFunctionCalls(event);
const usage = realtimeResponseUsage(event);

activeTurn.inputTokens += usage.inputTokens;
activeTurn.outputTokens += usage.outputTokens;
activeResponse.span.end();

if (calls.length === 0) {
  recordAiTurn({
    durationMs: performance.now() - activeTurn.startedAtMs,
    result: "success",
    provider: "openai",
    model: this.options.environment.OPENAI_REALTIME_MODEL,
    inputTokens: activeTurn.inputTokens,
    outputTokens: activeTurn.outputTokens,
  });
  activeTurn.span.end();
}

The surface-proposal tool is nested under that voice turn. Its structured model call now records provider/model attributes, token usage, estimated cost when the model exists in the checked-in price table, proposal validation, and persistence. Cancelled speech, provider errors, and socket failure close spans explicitly instead of leaking unfinished traces.

This was the lesson I could not get from an installation guide: instrument the lifecycle, not the endpoint that starts it.

The next fresh voice proposal produced the signals that were missing before:

This is not an empty dashboard: the selected window contains fallback, cost, worker, provider-duration, token, and complete agent-turn signals. The separate metric-histogram P95 panel still says No Data in this capture, while the trace-derived provider and agent-turn P95 panels are populated. I left that visible rather than presenting a cleaner but misleading screenshot.

The trace connects the live voice turn to proposal generation and exposes the failed provider request instead of hiding it behind the successful overall workflow.

Metrics and dashboards built around trust

The Node SDK exports traces, metrics, and selected correlated logs over OTLP/HTTP. Metrics use a 15-second periodic reader. I record:

AI turn volume and end-to-end latency
provider invocation duration and fallback
input/output tokens and estimated model cost
proposal operation, validation, decision, and undo outcomes
verifier and citation outcomes
queue wait and worker activity
canonical-safety invariant violations

Two idempotently provisioned dashboards contain eight panels each:

Sidechalk AI Command Center covers volume, P95 latency, provider duration, fallback, tokens, estimated cost, worker activity, and complete agent-turn duration.
Sidechalk Trust and Proposal Safety covers proposal operations/outcomes, verifier and citation activity, validation, invariant blocks, authoritative realtime delivery, and errors.

Four enabled rules watch provider reliability, learner-facing latency, model budget, and canonical safety. A tested local webhook accepts alert envelopes. I do not claim a production firing-and-recovery canary; this article describes my verified local stack.

Proposal creation, human decisions, validation, and shared delivery are monitored as separate trust stages.

“No Data” has three different meanings

This project produced three superficially similar empty states:

Expected absence: no fallback or invariant violation occurred. Empty is healthy evidence.
Missing instrumentation: in the pre-fix run, worker data arrived while the voice AI path stayed empty. The application path was invisible until I instrumented its sideband lifecycle.
Broken query infrastructure: a P95 query failed because my generated ClickHouse configuration did not load histogramQuantile correctly.

The third case was unusually specific. The generated executable-function configuration referenced the wrong filename pattern, and the UDF's quantile parameter used the wrong type. I corrected the local generated files, restarted only ClickHouse, and verified the function directly:

docker exec signoz-telemetrystore-clickhouse-0-0 \
  clickhouse-client --query \
  "select name from system.functions where name = 'histogramQuantile'"

The distinction matters. I should not manufacture a fallback event to make a panel attractive, but I also should not call an uninstrumented voice route healthy.

Reproducing my local path

I used Windows 11, Docker Desktop, WSL2, Node 24, and Foundry v0.2.14. Foundry generated the self-hosted SigNoZ stack from the checked-in casting.yaml:

foundryctl gauge -f casting.yaml
foundryctl cast -f casting.yaml

The standard container services use SigNoZ UI port 8080 and MCP port 8000. Those ports were already occupied on my machine, so my current host mappings are 18080 for the UI and 18000 for MCP. OTLP/HTTP remains 4318, and the Sidechalk verifier uses 8002. Check your own port owners instead of copying my remap blindly.

OTEL_ENABLED=true
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
OTEL_ENVIRONMENT=development
OTEL_LOGS_ENABLED=true
OTEL_CAPTURE_AI_CONTENT=false
OTEL_HASH_SALT=<private-random-value-at-least-16-characters>

SIGNOZ_UI_URL=http://localhost:18080
SIGNOZ_URL=http://localhost:18080
SIGNOZ_MCP_URL=http://127.0.0.1:18000/mcp
SIGNOZ_API_KEY=<service-account-key>
SIGNOZ_ALERT_CHANNEL=Local Webhook Receiver
OBSERVABILITY_ADMIN_USER_IDS=<my-Supabase-user-UUID>

From the repository root:

pnpm ports:preflight
docker compose up -d redis verifier
pnpm dev

pnpm signoz:provision --dry-run
pnpm signoz:provision
pnpm signoz:verify

The verifier checks the expected dashboard names, Query Builder version, panel IDs and counts, enabled alert routing, webhook-channel identity, and three representative telemetry queries. It deliberately failed when sidechalk.ai.turn.duration.bucket had last been seen four hours earlier. That failure was useful: provisioning existed, but fresh traffic had not proved the path.

After a new agent turn, this filter locates the workflow:

service.name = 'sidechalk-api' AND name = 'sidechalk.ai.turn'

Incident investigation without write access

Sidechalk also includes an Incident Copilot backed by the official SigNoZ MCP server. The MCP server can expose metrics, traces, logs, alerts, dashboards, and services; Sidechalk deliberately allowlists only read operations.

Each investigation is limited to four calls, 15 seconds per call, 45 seconds overall, and a maximum six-hour time window. Evidence cards retain the SigNoZ tool, time range, service, and deep link. Create, update, delete, and unrestricted execution tools are excluded.

A development-only Fault Lab can inject bounded provider delay/failure, verifier contradiction, or worker delay. Faults are labeled sidechalk.demo.fault.*, stay in memory, expire automatically, and are forbidden in production. Synthetic evidence must announce that it is synthetic.

The copilot queries read-only SigNoZ evidence and refuses to invent a root cause when no correlated fault or error is found.

Observability without copying the notebook

The most important telemetry decision was what not to attach to the AI-specific spans, metrics, and logs. Sidechalk's manual instrumentation excludes prompts, responses, handwriting, page images, audio, transcripts, authorization data, and raw user/board IDs. Manually recorded error spans keep only an error class and a fixed content-free message. IDs intentionally used for AI-workflow correlation are HMAC-hashed before export.

const forbiddenKey =
  /(?:prompt|response|content|body|text|audio|transcript|document|authorization|cookie|secret|api.?key|access.?token|email|user.?name)/iu;

export function safeAttributes(attributes: SafeAttributes = {}): Attributes {
  const result: Attributes = {};
  for (const [key, value] of Object.entries(attributes)) {
    if (value === undefined || forbiddenKey.test(key)) continue;
    result[key] = typeof value === "string" ? value.slice(0, 240) : value;
  }
  return result;
}

Those custom AI signals contain operation names, duration, provider/model, numeric token totals, proposal category/result, and safe correlations—not the learner's notebook content. I still treat generic auto-instrumentation as a separate privacy surface and avoid placing sensitive values in URLs or captured headers.

What I would tell myself before building this

Trace the path users actually take. A beautifully instrumented typed route did nothing for my live-voice demonstration.
Instrument product invariants. HTTP success cannot prove that an AI draft remained non-canonical.
Async agents outlive request spans. WebRTC and sideband tools need explicit context propagation and lifecycle closure.
Treat No Data as a diagnosis, not a cosmetic problem. Expected absence, missing telemetry, and query failure are different states.
Verify provisioning with live queries. A successful create/update response does not prove that panels can read real signals.
Keep investigation agents least-privileged. Evidence access does not require configuration write access.
Filter before export. Redaction after learner content reaches the backend is already too late.

Closing the loop

Sidechalk's promise is not merely that an AI can answer a physics question. It is that the agent can see authorized work, reason across a conversation, propose a spatial correction, wait for explicit approval, publish exactly one canonical update, and undo it safely.

SigNoZ made that promise testable. The temporary empty state showed where my first implementation was blind; the populated command center now shows that the real live-voice path is observable.

The AI can work on the page, but it cannot silently control the page—and now the path enforcing that rule is observable too.

From Exit 139 to HTTP 402: Following One Payment Failure Across SigNoz

Tyson — Sun, 19 Jul 2026 17:36:29 +0000

How I self-hosted SigNoz on WSL2, recovered a crashing telemetry pipeline, and followed one payment failure across metrics, traces, logs, and alerts.

Before I could debug a failed payment, I had to debug the observability system itself. The SigNoz interface loaded, but its telemetry path failed in stages: first ClickHouse Keeper, then the ingester and migrator, each exiting with code 139.

Once ingestion worked, the payment service showed a 5.08% error rate. A later failed request lasted 38 ms, contained 74 spans, and ended in HTTP 402 with an insufficient funds event; the same trace filter also powered a firing alert. The traffic was synthetic. What I had to work out was how to prove the failure: the useful error log lived on checkout rather than payment, and Logs Explorer rejected the trace expression I copied into it.

This article follows both investigations because the second was impossible until the first worked. The service, alert, and dashboard screenshots came from different rolling windows while the demo kept generating traffic. The trace and correlated logs below share the same trace ID.

My setup, briefly

Lab environment: Windows 11, Ubuntu 24.04.3 on WSL2, Docker Engine 29.6.2, Compose v5.3.1, Foundry v0.2.14, and about 7.6 GiB of RAM.

After verifying Docker with hello-world, I followed the current SigNoz Docker installation guide. My initial casting file used the Compose deployment mode and enabled the MCP server. I cast it from ~/signoz-lab:

curl -fsSL https://signoz.io/foundry.sh | bash
source ~/.bashrc

mkdir -p ~/signoz-lab
cd ~/signoz-lab
foundryctl cast -f casting.yaml

The first cast pulled the stack, but did not produce a working telemetry path.

Two setup details prevented later confusion. Windows-side localhost:8080 already belonged to Jenkins, so I opened SigNoz through the current IP address of my WSL2 instance. MCP appeared unhealthy because its container health check called a missing wget, although /readyz returned ok with HTTP 200.

I reproduced exit 139 outside the stack

ClickHouse Keeper entered a restart loop with exit code 139, and its logs were empty. There was no error text to search. Testing the exact image by itself was the first step that narrowed the problem:

docker run --rm clickhouse/clickhouse-keeper:25.12.5 --version
echo "exit=$?"

The image still exited 139 outside the Foundry-generated stack and without its usual dependencies. I tested other tags independently; 25.12.11 printed ClickHouse keeper version 25.12.11.4 and exited 0. The matching server image also ran.

Keeper was now healthy, but the ingester and schema migrator began printing Segmentation fault. The same isolation test reproduced exit 139 with signoz/signoz-otel-collector:latest. Running v0.144.5 with --version did not print a version—it returned unknown flag. That modest result still mattered: the process reached argument parsing instead of segfaulting at startup.

The isolated tests produced this diagnostic matrix:

clickhouse-keeper:25.12.5       → exit 139
clickhouse-keeper:25.12.11      → version 25.12.11.4, exit 0
signoz-otel-collector:latest     → exit 139
signoz-otel-collector:v0.144.5   → unknown flag: --version, exit 1

An exit status of 139 is conventionally read as 128 + 11, meaning the process received SIGSEGV. That identified the failure class, not its cause: on this machine, the crashing tags died before normal startup while the candidate tags did not.

Those tests narrowed the candidate tags; deployment still had to prove them. This is the casting file I used. These pins fixed this machine on July 18, 2026; they are not universal recommendations.

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  ingester:
    spec:
      image: signoz/signoz-otel-collector:v0.144.5
  telemetrystore:
    spec:
      image: clickhouse/clickhouse-server:25.12.11
  telemetrykeeper:
    spec:
      image: clickhouse/clickhouse-keeper:25.12.11
  mcp:
    spec:
      enabled: true

I saved the file, cast the deployment again, and verified the generated images and final process states:

foundryctl cast -f casting.yaml
docker compose -f pours/deployment/compose.yaml config --images
docker inspect --format \
  '{{.Name}} | image={{.Config.Image}} | status={{.State.Status}} | exit={{.State.ExitCode}}' \
  signoz-ingester-1 signoz-telemetrystore-migrator

The generated configuration used signoz/signoz-otel-collector:v0.144.5 for both the ingester and migrator. After initialization settled, the inspection returned:

/signoz-ingester-1 | image=signoz/signoz-otel-collector:v0.144.5 | status=running | exit=0
/signoz-telemetrystore-migrator | image=signoz/signoz-otel-collector:v0.144.5 | status=exited | exit=0

The pinned ingester stayed alive. The migrator finished with code 0, which I treated as successful completion in this run. The ingester.spec.image override follows Foundry's version-matched casting-file reference.

With ingestion working, I sent demo traffic

I cloned SigNoz's OpenTelemetry Demo Lite repository at its v0.0.6 tag and reduced the request rate to one request per second. The tagged repository's Compose file pulls signoz/opentelemetry-demo-lite:0.0.5, which is why both version numbers appeared in my terminal.

cd ~
git clone --branch v0.0.6 --depth 1 \
  https://github.com/SigNoz/opentelemetry-demo-lite.git
cd opentelemetry-demo-lite

WSL_IP=$(hostname -I | awk '{print $1}')
OTLP_ENDPOINT="${WSL_IP}:4317" \
OTLP_INSECURE=true \
RPS=1 \
docker compose up -d

The application first sent telemetry to the demo's local collector; that collector forwarded all three signals to SigNoz over OTLP/gRPC at ${WSL_IP}:4317. After five minutes, I checked the containers and the forwarding path:

docker inspect otel-collector \
  --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -E '^(OTLP_ENDPOINT|OTLP_INSECURE)='

docker compose ps -a

LOG_FILE=/tmp/otel-collector-last5m.log
docker compose logs --since 5m otel-collector > "$LOG_FILE" 2>&1
echo "LOG_COMMAND_EXIT=$?"

grep -Ein 'error|failed|refused|unavailable|retry' \
  "$LOG_FILE"

grep -Ein \
  'export.*(error|fail)|failed.*export|sending_queue|retry.*(export|send)|connection refused|rpc error|context deadline exceeded|service unavailable' \
  "$LOG_FILE"
echo "EXPORT_SCAN_EXIT=$?"

grep -Ein \
  'Starting GRPC server|Starting HTTP server|Everything is ready' \
  "$LOG_FILE"

In that run, OTLP_ENDPOINT resolved to 172.18.31.214:4317, insecure mode was enabled, and all three demo containers were up; the mock application and Redis were healthy. Log retrieval exited 0. The broad scan found one non-export warning: the configured Docker resource detector could not access /var/run/docker.sock. That is consistent with the version-matched resource detector documentation, which says the Docker detector queries the daemon and requires its socket. The collector continued, detected its host and OS, started OTLP receivers on 4317 and 4318, and reported:

Everything is ready. Begin running and processing data.

The focused scan returned EXPORT_SCAN_EXIT=1: grep found none of those exporter-failure patterns in the five-minute window. That was not proof of storage. SigNoz supplied that proof by showing active logs, traces, and metrics ingestion across 14 services.

The payment service had a 5.08% error rate

In the selected 30-minute window, the payment service's charge operation showed 1,790 calls and a 5.08% error rate. Its P50, P95, and P99 latencies were 0.81 ms, 1.40 ms, and 2.22 ms. The standout signal in that snapshot was failed work, not a long-running request.

The service view told me where to look, but not why the charges were failing.

In a later window of the same continuing synthetic traffic, I moved to Traces Explorer and used:

service.name = 'payment' AND name = 'charge' AND has_error = true

One result opened a distributed checkout path:

trace ID 51b217b6cf309cd08cc87224e8712f57
timestamp July 19, 2026, 21:36:08 IST
38 ms total duration
74 spans
two error spans
an upstream checkout HTTP POST with status 402
a payment / charge span lasting 0.56 ms

The trace located the failure. The selected span's events explained it: Payment failed: insufficient funds.

Here, HTTP 402 was the boundary outcome, the span event held the domain reason, and the checkout log later supplied the caller's interpretation.

The checkout client span recorded the HTTP 402 boundary-level failure

The child payment span supplied the domain reason: insufficient funds.

The status code described the response at the service boundary; the span event supplied the domain reason.

The log was not where I first looked

During the investigation, an earlier payment span's Logs tab showed no attached records. I also copied its trace expression into Logs Explorer, where SigNoz rejected it with:

key 'name' not found
key 'has_error' not found

In this dataset, name and has_error were valid trace fields but were absent from the log schema. I had crossed a signal boundary without changing schemas. Trace context was the bridge: SigNoz correlates the signals through trace_id and span_id, as described in its logs-and-traces correlation guide.

For the 21:36:08 request shown above, I carried its trace context into Logs Explorer with trace_id = '51b217b6cf309cd08cc87224e8712f57'. That found records from the distributed request, including ChargeCard failed and Payment failed. The relevant error came from the checkout service and shared the trace's trace_id; its span_id identified the checkout span that emitted it:

{
  "body": "Payment failed",
  "attributes": {
    "error": "payment service returned 402"
  },
  "resources": {
    "service.name": "checkout",
    "deployment.environment": "demo"
  },
  "severity_text": "ERROR",
  "span_id": "62c755a937ef91d4",
  "trace_id": "51b217b6cf309cd08cc87224e8712f57"
}

Filtering the request by trace ID exposed what checkout recorded.

That distinction is why this investigation was more useful than a feature tour. Metrics revealed a pattern. The trace reconstructed one request. A span event supplied the reason. A log showed how the failure was recorded by another service. The signals agreed without pretending they were interchangeable.

I reused the query instead of rebuilding it

I saved the filtered trace view and reused its expression for the dashboard and trace-based alert Payment Charge Failures. The rule counts matches over a rolling five-minute window and fires at critical when any occur, using SigNoz's standard trace-based alert workflow.

For the required notification channel, I ran a temporary receiver on the generated Docker network:

docker run -d \
  --name signoz-webhook-receiver \
  --network signoz-network \
  -p 127.0.0.1:9001:8000 \
  python:3.12-alpine \
  python -u -c 'from http.server import BaseHTTPRequestHandler,HTTPServer; H=type("H",(BaseHTTPRequestHandler,),{"do_POST":lambda self:(lambda data:(print(data.decode("utf-8","replace"),flush=True),self.send_response(200),self.end_headers(),self.wfile.write(b"ok")))(self.rfile.read(int(self.headers.get("Content-Length","0"))))}); HTTPServer(("0.0.0.0",8000),H).serve_forever()'

My deployment called the network signoz-network; another Compose project may differ. I gave the receiver no restart policy.

Before configuring the channel, I sent it JSON directly:

curl -i http://localhost:9001 \
  -H 'Content-Type: application/json' \
  -d '{"test":"working-after-restart"}'

docker logs --tail 20 signoz-webhook-receiver

The POST returned HTTP 200 and appeared in the logs. I set the channel URL to http://signoz-webhook-receiver:8000, recorded a UTC start time, clicked Test Notification, and inspected only newer logs:

TEST_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
# Click "Test Notification" in SigNoz.
docker logs --since "$TEST_START" --timestamps \
  signoz-webhook-receiver

The test returned HTTP 200 and produced:

{"receiver":"Local Webhook Receiver","status":"firing",
 "alerts":[{"labels":{"alertname":"Test Alert (Local Webhook Receiver)",
 "severity":"critical"}}]}
"POST / HTTP/1.1" 200 -

The alert screenshot predates the selected 21:36 trace because the rule had already fired on earlier occurrences of the same repeating synthetic condition. The payload proves the channel test worked; the firing badge proves the real rule crossed its threshold. I did not capture an actual-rule webhook payload, so I will not claim one.

I then removed the temporary receiver:

docker rm -f signoz-webhook-receiver

The same filter used during debugging now evaluated automatically.

I also expanded the failed-count graph into a Demo Observability payment-health dashboard:

Failed Payment Charges Over Time applies the error filter and plots count().
Total Payment Charge Operations uses the same filter without has_error = true and displays count() as a number. In the captured rolling window, it showed 1,797.
Payment Charge Error Rate Over Time calculates (A/B)*100, where A is failed charges and B is all charges. I hid A and B and formatted the result as a percentage.
P95 Payment Charge Latency uses p95(duration_nano); with my panel's unit configuration, SigNoz displayed it in milliseconds.

I left Fill gaps disabled on the rate and latency panels because a missing interval is not automatically a measured zero, especially for a rate with a denominator. The service view showed 1,790 calls and the dashboard showed 1,797 because they were captured from different rolling 30-minute windows on July 19. Those values are not directly comparable; the dashboard is also bucketed, so its brief 60% peak is not a whole-window error rate.

The four panels answer different questions: when failures occurred, how much traffic was evaluated, what proportion failed, and whether latency changed.

What I would keep from this lab

Verify beyond badges. SigNoz served pages while its ingester restarted, and an empty error scan was not enough. Receiving-side telemetry was stronger evidence.
Reduce crashes. Testing the exact image separated startup failure from Compose dependencies and configuration.
Respect signal schemas. Trace filters do not automatically apply to logs; trace_id was the bridge.
Reuse queries and qualify workarounds. One filter became a view, alert, and dashboard, while the pins remained explicitly machine-specific.

Tested on Windows 11 with WSL2 on July 18–19, 2026. Image tags are mutable; check the current SigNoz self-hosting documentation before reusing these pins.

The lab began with a working UI alongside exit 139 crashes and ended with an HTTP 402 I could trace into checkout logs and monitor afterward. That cross-signal context is what I would use SigNoz for again.