DEV Community

Cover image for From Exit 139 to HTTP 402: Following One Payment Failure Across SigNoz
Tyson
Tyson

Posted on

From Exit 139 to HTTP 402: Following One Payment Failure Across SigNoz

#signoz #opentelemetry #devops #observability

How I self-hosted SigNoz on WSL2, recovered a crashing telemetry pipeline, and followed one payment failure across metrics, traces, logs, and alerts.

Before I could debug a failed payment, I had to debug the observability system itself. The SigNoz interface loaded, but its telemetry path failed in stages: first ClickHouse Keeper, then the ingester and migrator, each exiting with code 139.

Once ingestion worked, the payment service showed a 5.08% error rate. A later failed request lasted 38 ms, contained 74 spans, and ended in HTTP 402 with an insufficient funds event; the same trace filter also powered a firing alert. The traffic was synthetic. What I had to work out was how to prove the failure: the useful error log lived on checkout rather than payment, and Logs Explorer rejected the trace expression I copied into it.

This article follows both investigations because the second was impossible until the first worked. The service, alert, and dashboard screenshots came from different rolling windows while the demo kept generating traffic. The trace and correlated logs below share the same trace ID.

My setup, briefly

Lab environment: Windows 11, Ubuntu 24.04.3 on WSL2, Docker Engine 29.6.2, Compose v5.3.1, Foundry v0.2.14, and about 7.6 GiB of RAM.

After verifying Docker with hello-world, I followed the current SigNoz Docker installation guide. My initial casting file used the Compose deployment mode and enabled the MCP server. I cast it from ~/signoz-lab:

curl -fsSL https://signoz.io/foundry.sh | bash
source ~/.bashrc

mkdir -p ~/signoz-lab
cd ~/signoz-lab
foundryctl cast -f casting.yaml
Enter fullscreen mode Exit fullscreen mode

The first cast pulled the stack, but did not produce a working telemetry path.

Two setup details prevented later confusion. Windows-side localhost:8080 already belonged to Jenkins, so I opened SigNoz through the current IP address of my WSL2 instance. MCP appeared unhealthy because its container health check called a missing wget, although /readyz returned ok with HTTP 200.

I reproduced exit 139 outside the stack

ClickHouse Keeper entered a restart loop with exit code 139, and its logs were empty. There was no error text to search. Testing the exact image by itself was the first step that narrowed the problem:

docker run --rm clickhouse/clickhouse-keeper:25.12.5 --version
echo "exit=$?"
Enter fullscreen mode Exit fullscreen mode

The image still exited 139 outside the Foundry-generated stack and without its usual dependencies. I tested other tags independently; 25.12.11 printed ClickHouse keeper version 25.12.11.4 and exited 0. The matching server image also ran.

Keeper was now healthy, but the ingester and schema migrator began printing Segmentation fault. The same isolation test reproduced exit 139 with signoz/signoz-otel-collector:latest. Running v0.144.5 with --version did not print a version—it returned unknown flag. That modest result still mattered: the process reached argument parsing instead of segfaulting at startup.

The isolated tests produced this diagnostic matrix:

clickhouse-keeper:25.12.5       → exit 139
clickhouse-keeper:25.12.11      → version 25.12.11.4, exit 0
signoz-otel-collector:latest     → exit 139
signoz-otel-collector:v0.144.5   → unknown flag: --version, exit 1
Enter fullscreen mode Exit fullscreen mode

An exit status of 139 is conventionally read as 128 + 11, meaning the process received SIGSEGV. That identified the failure class, not its cause: on this machine, the crashing tags died before normal startup while the candidate tags did not.

Those tests narrowed the candidate tags; deployment still had to prove them. This is the casting file I used. These pins fixed this machine on July 18, 2026; they are not universal recommendations.

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  ingester:
    spec:
      image: signoz/signoz-otel-collector:v0.144.5
  telemetrystore:
    spec:
      image: clickhouse/clickhouse-server:25.12.11
  telemetrykeeper:
    spec:
      image: clickhouse/clickhouse-keeper:25.12.11
  mcp:
    spec:
      enabled: true
Enter fullscreen mode Exit fullscreen mode

I saved the file, cast the deployment again, and verified the generated images and final process states:

foundryctl cast -f casting.yaml
docker compose -f pours/deployment/compose.yaml config --images
docker inspect --format \
  '{{.Name}} | image={{.Config.Image}} | status={{.State.Status}} | exit={{.State.ExitCode}}' \
  signoz-ingester-1 signoz-telemetrystore-migrator
Enter fullscreen mode Exit fullscreen mode

The generated configuration used signoz/signoz-otel-collector:v0.144.5 for both the ingester and migrator. After initialization settled, the inspection returned:

/signoz-ingester-1 | image=signoz/signoz-otel-collector:v0.144.5 | status=running | exit=0
/signoz-telemetrystore-migrator | image=signoz/signoz-otel-collector:v0.144.5 | status=exited | exit=0
Enter fullscreen mode Exit fullscreen mode

The pinned ingester stayed alive. The migrator finished with code 0, which I treated as successful completion in this run. The ingester.spec.image override follows Foundry's version-matched casting-file reference.

With ingestion working, I sent demo traffic

I cloned SigNoz's OpenTelemetry Demo Lite repository at its v0.0.6 tag and reduced the request rate to one request per second. The tagged repository's Compose file pulls signoz/opentelemetry-demo-lite:0.0.5, which is why both version numbers appeared in my terminal.

cd ~
git clone --branch v0.0.6 --depth 1 \
  https://github.com/SigNoz/opentelemetry-demo-lite.git
cd opentelemetry-demo-lite

WSL_IP=$(hostname -I | awk '{print $1}')
OTLP_ENDPOINT="${WSL_IP}:4317" \
OTLP_INSECURE=true \
RPS=1 \
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

The application first sent telemetry to the demo's local collector; that collector forwarded all three signals to SigNoz over OTLP/gRPC at ${WSL_IP}:4317. After five minutes, I checked the containers and the forwarding path:

docker inspect otel-collector \
  --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -E '^(OTLP_ENDPOINT|OTLP_INSECURE)='

docker compose ps -a

LOG_FILE=/tmp/otel-collector-last5m.log
docker compose logs --since 5m otel-collector > "$LOG_FILE" 2>&1
echo "LOG_COMMAND_EXIT=$?"

grep -Ein 'error|failed|refused|unavailable|retry' \
  "$LOG_FILE"

grep -Ein \
  'export.*(error|fail)|failed.*export|sending_queue|retry.*(export|send)|connection refused|rpc error|context deadline exceeded|service unavailable' \
  "$LOG_FILE"
echo "EXPORT_SCAN_EXIT=$?"

grep -Ein \
  'Starting GRPC server|Starting HTTP server|Everything is ready' \
  "$LOG_FILE"
Enter fullscreen mode Exit fullscreen mode

In that run, OTLP_ENDPOINT resolved to 172.18.31.214:4317, insecure mode was enabled, and all three demo containers were up; the mock application and Redis were healthy. Log retrieval exited 0. The broad scan found one non-export warning: the configured Docker resource detector could not access /var/run/docker.sock. That is consistent with the version-matched resource detector documentation, which says the Docker detector queries the daemon and requires its socket. The collector continued, detected its host and OS, started OTLP receivers on 4317 and 4318, and reported:

Everything is ready. Begin running and processing data.
Enter fullscreen mode Exit fullscreen mode

The focused scan returned EXPORT_SCAN_EXIT=1: grep found none of those exporter-failure patterns in the five-minute window. That was not proof of storage. SigNoz supplied that proof by showing active logs, traces, and metrics ingestion across 14 services.

The payment service had a 5.08% error rate

In the selected 30-minute window, the payment service's charge operation showed 1,790 calls and a 5.08% error rate. Its P50, P95, and P99 latencies were 0.81 ms, 1.40 ms, and 2.22 ms. The standout signal in that snapshot was failed work, not a long-running request.

SigNoz payment metrics showing 1,790 charge calls and a 5.08 percent error rate
The service view told me where to look, but not why the charges were failing.

In a later window of the same continuing synthetic traffic, I moved to Traces Explorer and used:

service.name = 'payment' AND name = 'charge' AND has_error = true
Enter fullscreen mode Exit fullscreen mode

One result opened a distributed checkout path:

  • trace ID 51b217b6cf309cd08cc87224e8712f57
  • timestamp July 19, 2026, 21:36:08 IST
  • 38 ms total duration
  • 74 spans
  • two error spans
  • an upstream checkout HTTP POST with status 402
  • a payment / charge span lasting 0.56 ms

The trace located the failure. The selected span's events explained it: Payment failed: insufficient funds.

Here, HTTP 402 was the boundary outcome, the span event held the domain reason, and the checkout log later supplied the caller's interpretation.

SigNoz trace showing the checkout HTTP POST span returning HTTP 402
The checkout client span recorded the HTTP 402 boundary-level failure

SigNoz payment charge span showing an insufficient-funds exception
The child payment span supplied the domain reason: insufficient funds.

The status code described the response at the service boundary; the span event supplied the domain reason.

The log was not where I first looked

During the investigation, an earlier payment span's Logs tab showed no attached records. I also copied its trace expression into Logs Explorer, where SigNoz rejected it with:

key 'name' not found
key 'has_error' not found
Enter fullscreen mode Exit fullscreen mode

In this dataset, name and has_error were valid trace fields but were absent from the log schema. I had crossed a signal boundary without changing schemas. Trace context was the bridge: SigNoz correlates the signals through trace_id and span_id, as described in its logs-and-traces correlation guide.

For the 21:36:08 request shown above, I carried its trace context into Logs Explorer with trace_id = '51b217b6cf309cd08cc87224e8712f57'. That found records from the distributed request, including ChargeCard failed and Payment failed. The relevant error came from the checkout service and shared the trace's trace_id; its span_id identified the checkout span that emitted it:

{
  "body": "Payment failed",
  "attributes": {
    "error": "payment service returned 402"
  },
  "resources": {
    "service.name": "checkout",
    "deployment.environment": "demo"
  },
  "severity_text": "ERROR",
  "span_id": "62c755a937ef91d4",
  "trace_id": "51b217b6cf309cd08cc87224e8712f57"
}
Enter fullscreen mode Exit fullscreen mode

SigNoz Logs Explorer filtered by trace ID showing Payment failed and ChargeCard failed
Filtering the request by trace ID exposed what checkout recorded.

That distinction is why this investigation was more useful than a feature tour. Metrics revealed a pattern. The trace reconstructed one request. A span event supplied the reason. A log showed how the failure was recorded by another service. The signals agreed without pretending they were interchangeable.

I reused the query instead of rebuilding it

I saved the filtered trace view and reused its expression for the dashboard and trace-based alert Payment Charge Failures. The rule counts matches over a rolling five-minute window and fires at critical when any occur, using SigNoz's standard trace-based alert workflow.

For the required notification channel, I ran a temporary receiver on the generated Docker network:

docker run -d \
  --name signoz-webhook-receiver \
  --network signoz-network \
  -p 127.0.0.1:9001:8000 \
  python:3.12-alpine \
  python -u -c 'from http.server import BaseHTTPRequestHandler,HTTPServer; H=type("H",(BaseHTTPRequestHandler,),{"do_POST":lambda self:(lambda data:(print(data.decode("utf-8","replace"),flush=True),self.send_response(200),self.end_headers(),self.wfile.write(b"ok")))(self.rfile.read(int(self.headers.get("Content-Length","0"))))}); HTTPServer(("0.0.0.0",8000),H).serve_forever()'
Enter fullscreen mode Exit fullscreen mode

My deployment called the network signoz-network; another Compose project may differ. I gave the receiver no restart policy.

Before configuring the channel, I sent it JSON directly:

curl -i http://localhost:9001 \
  -H 'Content-Type: application/json' \
  -d '{"test":"working-after-restart"}'

docker logs --tail 20 signoz-webhook-receiver
Enter fullscreen mode Exit fullscreen mode

The POST returned HTTP 200 and appeared in the logs. I set the channel URL to http://signoz-webhook-receiver:8000, recorded a UTC start time, clicked Test Notification, and inspected only newer logs:

TEST_START=$(date -u +%Y-%m-%dT%H:%M:%SZ)
# Click "Test Notification" in SigNoz.
docker logs --since "$TEST_START" --timestamps \
  signoz-webhook-receiver
Enter fullscreen mode Exit fullscreen mode

The test returned HTTP 200 and produced:

{"receiver":"Local Webhook Receiver","status":"firing",
 "alerts":[{"labels":{"alertname":"Test Alert (Local Webhook Receiver)",
 "severity":"critical"}}]}
"POST / HTTP/1.1" 200 -
Enter fullscreen mode Exit fullscreen mode

The alert screenshot predates the selected 21:36 trace because the rule had already fired on earlier occurrences of the same repeating synthetic condition. The payload proves the channel test worked; the firing badge proves the real rule crossed its threshold. I did not capture an actual-rule webhook payload, so I will not claim one.

I then removed the temporary receiver:

docker rm -f signoz-webhook-receiver
Enter fullscreen mode Exit fullscreen mode

SigNoz Payment Charge Failures alert firing at critical severity
The same filter used during debugging now evaluated automatically.

I also expanded the failed-count graph into a Demo Observability payment-health dashboard:

  • Failed Payment Charges Over Time applies the error filter and plots count().
  • Total Payment Charge Operations uses the same filter without has_error = true and displays count() as a number. In the captured rolling window, it showed 1,797.
  • Payment Charge Error Rate Over Time calculates (A/B)*100, where A is failed charges and B is all charges. I hid A and B and formatted the result as a percentage.
  • P95 Payment Charge Latency uses p95(duration_nano); with my panel's unit configuration, SigNoz displayed it in milliseconds.

I left Fill gaps disabled on the rate and latency panels because a missing interval is not automatically a measured zero, especially for a rate with a denominator. The service view showed 1,790 calls and the dashboard showed 1,797 because they were captured from different rolling 30-minute windows on July 19. Those values are not directly comparable; the dashboard is also bucketed, so its brief 60% peak is not a whole-window error rate.

SigNoz payment-health dashboard showing failed charges, operations, error rate, and P95 latency
The four panels answer different questions: when failures occurred, how much traffic was evaluated, what proportion failed, and whether latency changed.

What I would keep from this lab

  1. Verify beyond badges. SigNoz served pages while its ingester restarted, and an empty error scan was not enough. Receiving-side telemetry was stronger evidence.
  2. Reduce crashes. Testing the exact image separated startup failure from Compose dependencies and configuration.
  3. Respect signal schemas. Trace filters do not automatically apply to logs; trace_id was the bridge.
  4. Reuse queries and qualify workarounds. One filter became a view, alert, and dashboard, while the pins remained explicitly machine-specific.

Tested on Windows 11 with WSL2 on July 18–19, 2026. Image tags are mutable; check the current SigNoz self-hosting documentation before reusing these pins.

The lab began with a working UI alongside exit 139 crashes and ended with an HTTP 402 I could trace into checkout logs and monitor afterward. That cross-signal context is what I would use SigNoz for again.

Top comments (0)