DEV Community

Cover image for How I Caught a Silent Payment Outage Across 7 Microservices Using SigNoz & OpenTelemetry
vaishnav
vaishnav

Posted on Edited on

How I Caught a Silent Payment Outage Across 7 Microservices Using SigNoz & OpenTelemetry

#signoz #wemakedev

WeMakeDevs x SigNoz: "Agents of SigNoz" Hackathon 2026

Silent failures and healthy logs

Before configuring OpenTelemetry and SigNoz, my application appeared perfectly healthy. Health checks passed, and the API responded normally.

Underneath, however, a batch of orders disappeared every few minutes. The system did not throw exceptions or print error messages to stdout. The revenue loss went entirely unnoticed.

I wrote the openSlice simulator to investigate this problem. I wanted to see how a payment provider failure impacts on 7 Microservices e-commerce system, and if I could expose these silent issues in less than 10 minutes using SigNoz and OpenTelemetry.

The objective was to set up OpenTelemetry in a Python application, export the data to SigNoz, and use trace waterfalls and error metrics to diagnose incidents.

The openSlice simulator

The openSlice tool simulates a Python e-commerce backend. It uses threads and latency models instead of real databases or HTTP calls, generating OTLP telemetry without running a full microservices cluster.

https://github.com/thisisvaishnav/openSlice

Service topology 

api-gateway  ->  (root span)
  ├──->  auth-service  ->  product-catalog
  ├──->  cart-service
  ├──->  order-service  ->  payment-service (failing)  ->  notification
  └──->  product-catalog
Enter fullscreen mode Exit fullscreen mode

Each service runs in its own thread with a separate TracerProvider and MeterProvider to mimic independent microservices. The services pass context using W3C Trace Context headers through OpenTelemetry's inject and extract functions, allowing SigNoz to construct the full trace waterfall.

The simulator's chaos engine randomly triggers one of four incident modes in the payment service:

Incident Behavior
DEGRADED Latency increases to 2-5 seconds, 30% error rate
PARTIAL_OUTAGE 60% of requests fail
PROVIDER_FAILURE All Stripe requests fail; other payment methods remain healthy
MEMORY_PRESSURE Latency increases linearly over 45 seconds to 3,000ms

When a provider failure occurs, the order service also fails because it depends on the payment service to complete transactions. This cascading failure is what we want SigNoz to help us detect.

Running SigNoz locally

You can run SigNoz using Docker Desktop without a cloud account or a Kubernetes cluster.

1. Start SigNoz

Clone the SigNoz repository and start the services using Docker Compose. This runs ClickHouse, the OTel Collector, and the SigNoz UI.

git clone https://github.com/SigNoz/signoz.git
cd signoz/deploy
docker compose up -d
Enter fullscreen mode Exit fullscreen mode

2. Install Python dependencies

Install the OpenTelemetry SDK, OTLP HTTP exporter, and protobuf packages.

# requirements.txt
opentelemetry-sdk==1.34.1
opentelemetry-exporter-otlp-proto-http==1.34.1
opentelemetry-api==1.34.1
protobuf==5.29.4
Enter fullscreen mode Exit fullscreen mode 
cd ~/openSlice
pip install -r requirements.txt
Enter fullscreen mode Exit fullscreen mode

3. Start the simulator

Run the simulator to start sending traces and metrics to your local SigNoz instance.

# Chaos mode is enabled by default
python platform_simulator.py

# To disable incident injection:
python platform_simulator.py --no-chaos

# To point to a remote SigNoz endpoint:
python platform_simulator.py --endpoint http://your-signoz:4318
Enter fullscreen mode Exit fullscreen mode

Verifying the simulator execution

After starting the simulator, the terminal displays a live status table that refreshes every 10 seconds. This table shows which services are active and whether chaos injection is running.

Figure 1: Terminal output from the simulator showing service health status.

The SigNoz UI is accessible at http://localhost:8080. It takes about a minute for the services to show up in the Services tab. You do not need to register them manually.

How to instrument the simulator code

The simulator uses the ServiceThread class to represent each service. It initializes the OpenTelemetry providers and wraps requests in spans.

1. Separate TracerProviders for each service

SigNoz separates services using the service.name attribute on the OpenTelemetry resource. If your services share a TracerProvider, SigNoz will group them together in the dashboard. Each service must have its own provider with a distinct resource:

from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

self.resource = Resource.create({
    "service.name": self.service_name,
    "service.version": "1.0.0",
    "deployment.environment": "production",
})

trace_exporter = OTLPSpanExporter(endpoint=f"{endpoint}/v1/traces")
self.tracer_provider = TracerProvider(resource=self.resource)
self.tracer_provider.add_span_processor(BatchSpanProcessor(trace_exporter))
self.tracer = self.tracer_provider.get_tracer(self.service_name)
Enter fullscreen mode Exit fullscreen mode

2. Distributing trace context

The API gateway creates the root span and injects the W3C traceparent header into the request headers. Downstream services extract this header, linking their spans as children of the gateway's span.

from opentelemetry.propagate import inject, extract

# API Gateway: starts the root span and injects context
with self.tracer.start_as_current_span(
    f"{self.service_name}/handle_request", kind=SpanKind.SERVER
) as gateway_span:
    for downstream in self.downstream:
        headers = {}
        inject(headers)
        downstream._handle_downstream(headers)

# Downstream service: extracts the context and creates a child span
def _handle_downstream(self, parent_headers: dict):
    ctx = extract(parent_headers)
    with self.tracer.start_as_current_span(
        f"{self.service_name}/process",
        context=ctx,
        kind=SpanKind.SERVER,
    ) as span:
        # Process the request
Enter fullscreen mode Exit fullscreen mode

Note: If you do not pass context=ctx during downstream span creation, the spans will show up in SigNoz as separate root traces rather than a linked parent-child waterfall.

3. Setting up metrics

Each service collects request counts, error counts, latency, and rolling error rates.

# OpenTelemetry metrics instruments
self.request_counter   = self.meter.create_counter(f"{self.service_name}.requests", unit="1")
self.error_counter     = self.meter.create_counter(f"{self.service_name}.errors",   unit="1")
self.latency_histogram = self.meter.create_histogram(f"{self.service_name}.latency", unit="ms")

# Observable gauge to calculate and emit the rolling error rate
def make_callback(svc_name, s):
    def callback(options):
        snap = s.snapshot()
        err_rate = snap["err_count"] / max(snap["total"], 1)
        yield metrics.Observation(err_rate, {"service.name": svc_name})
    return callback

st.meter.create_observable_gauge(
    name=f"{st.service_name}.error_rate",
    callbacks=[make_callback(st.service_name, svc_stats)],
    unit="1",
)
Enter fullscreen mode Exit fullscreen mode

4. Adding custom span attributes for chaos events

During simulated incidents, the simulator adds the active incident type to the span. This allows you to filter and query traces in SigNoz directly.

# Add the incident type to the span
if self.service_name == "payment-service" and self.incident_state.active:
    span.set_attribute("incident.type", self.incident_state.incident_type)

# Mark Stripe requests as failed during a provider failure
if (self.service_name == "payment-service"
        and self.incident_state.incident_type == "PROVIDER_FAILURE"
        and self._is_stripe_request()):
    span.set_attribute("payment.method", "stripe")
    return True
Enter fullscreen mode Exit fullscreen mode

Viewing telemetry in SigNoz

When the simulator starts running, the SigNoz dashboard visualizes the telemetry.

Services dashboard

The Services page lists all seven simulated services. SigNoz maps service dependencies automatically. When errors occur in the payment service, the UI highlights the service and shows how the issues affect the upstream order service.

Trace explorer and request waterfalls

To view trace details, navigate to the Trace Explorer, filter by service: api-gateway, and select a trace.

SigNoz connects the parent and child spans. You can see the api-gateway make concurrent requests to downstream services, with the order service calling the payment service. If a provider failure happens, the payment service span highlights in red with an error status.

Figure 2: SigNoz Trace Explorer displaying a failed payment service span.

Metrics dashboard

You can build custom dashboards in SigNoz to track metrics. A two-panel dashboard can monitor P99 latency and error rates across services.

Metric Healthy During incident
P99 Latency (payment-service) ~400ms 4,500ms
Error Rate (payment-service) ~5% 95%
Error Rate (order-service) ~4% 20% (cascading failure)
Other services baseline unchanged

Figure 3: Dashboard monitoring payment service latency and cascading error rates in the order service.

Querying trace attributes

Debugging a payment service failure usually requires searching through application logs and matching timestamps to find the transaction details.

In SigNoz, clicking on a failed span displays its custom attributes in the side panel. For example, during a Stripe outage, the span contains the attributes payment.method: stripe and incident.type: PROVIDER_FAILURE. This allows you to identify the specific error source without searching through log files.

Lessons learned

  1. Each service needs its own TracerProvider. Sharing a TracerProvider combines multiple services into one entity in SigNoz. The OpenTelemetry specification identifies services by the resource object, making individual providers necessary.
  2. Context propagation is required. If you do not pass the context to child spans, the parent-child relationships break. The trace waterfall then displays as a flat list of unrelated spans.
  3. Use span attributes over text logs. Adding details like the incident type and payment method as attributes allows you to filter and aggregate traces directly in SigNoz.
  4. Focus on key metrics. Tracking P99 latency and error rates is usually sufficient to identify and locate failures. Adding more dashboards often increases noise rather than clarity.

Summary

Using SigNoz, the simulated payment outage was visible in seconds. The UI highlighted the latency spike, the elevated error rate, and the specific failure attributes on the traces.

Setting up SigNoz via Docker and configuring the simulator took under 30 minutes. This configuration provides tracing and metrics out of the box.

If you are developing Python services, implementing OpenTelemetry helps capture silent failures that standard log files might miss.

To test the setup, clone the openSlice repository, start SigNoz, and run the simulator to view the injected failures in the dashboard.

Resources

Written for the WeMakeDevs and SigNoz "Agents of SigNoz" Hackathon 2026 | openSlice project

Top comments (1)

Collapse
 
vaishnavxblog profile image
vaishnav •

please give feedback about this blog