From a Payment Timeout to the Root Cause: Building SentinelOps with SigNoz and OpenTelemetry

Pranjal Verma — Sun, 26 Jul 2026 18:01:30 +0000

Modern applications rarely fail in one obvious place. A request may enter through one service, communicate with several others, and finally fail because one downstream dependency became slow or returned an error.

During the SigNoz hackathon, we wanted to explore a simple question:

Can we go beyond monitoring failures and build a system that automatically turns observability data into actionable incidents?

That idea became SentinelOps.

SentinelOps is an intelligent observability and incident-response platform built on top of SigNoz. It uses OpenTelemetry to collect telemetry from distributed services, SigNoz to observe and query that telemetry, a rule engine to detect abnormal behavior, and SigNoz MCP for AI-assisted incident investigation.

The Problem We Wanted to Solve

Imagine an order request failing with a 504 Gateway Timeout.

Knowing that the request failed is useful, but as a developer, the questions that follow are more important:

Which service caused the failure?
Was it an application error or a latency problem?
Which downstream call became slow?
What logs were generated around the failure?
Are similar failures happening repeatedly?

Normally, answering these questions means jumping between logs, traces, metrics, dashboards, and alerts.

We wanted SentinelOps to connect these pieces into one incident workflow.

Our Architecture

For the demo environment, we created four Python/FastAPI microservices:

Auth Service
     │
     ▼
Order Service ─────► Product Service
     │
     ▼
Payment Service

Each service has a different responsibility:

Auth Service — authentication and JWT validation
Product Service — product and inventory information
Order Service — orchestrates the complete order workflow
Payment Service — processes payments and simulates failures

The services are instrumented using OpenTelemetry and export telemetry to SigNoz.

On top of them, SentinelOps has a Node.js/Express backend responsible for querying observability data, evaluating monitoring rules, managing incidents, and providing data to the frontend.

MongoDB stores rules, incidents, configuration, and analysis results.

Overall flow

                    SentinelOps Dashboard
                             │
                             ▼
                    Node.js / Express
                       Backend + Rules
                         │         │
                         ▼         ▼
                      MongoDB    SigNoz
                                   ▲
                                   │ OTLP
              ┌────────────────────┼────────────────────┐
              │                    │                    │
           Auth                Product              Order
                                                     │
                                                     ▼
                                                  Payment

Instrumenting Our Services with OpenTelemetry

Our first major task was getting telemetry from the Python services into SigNoz.

We configured each service with OpenTelemetry environment variables.

For example:

export OTEL_RESOURCE_ATTRIBUTES="service.name=auth-service,deployment.environment=hackathon"

export OTEL_EXPORTER_OTLP_ENDPOINT="http://<SIGNOZ_HOST>:4318"

export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

export OTEL_TRACES_EXPORTER="otlp"

export OTEL_LOGS_EXPORTER="otlp"

export OTEL_METRICS_EXPORTER="otlp"

The applications were then started through OpenTelemetry instrumentation.

For FastAPI/Uvicorn, this looked like:

opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000

We repeated the setup for:

auth-service
product-service
order-service
payment-service

This gave SigNoz visibility across the complete request chain.

Our First Problem: Nothing Was Reaching SigNoz

One of the most useful parts of the hackathon was that the setup didn't work immediately.

Initially, we tried reaching the OTLP HTTP receiver:

curl --connect-timeout 5 -v http://<SIGNOZ_HOST>:4318/

Instead of connecting, we received:

connect to <SIGNOZ_HOST> port 4318 failed:
Connection timed out

curl: (28) Connection timeout after 5001 ms

Our application configuration was correct, but the OpenTelemetry receiver wasn't externally reachable.

After allowing access to port 4318, we tested again:

Connected to <SIGNOZ_HOST> port 4318

HTTP/1.1 404 Not Found

At first, a 404 might look like another error.

In this case, however, it was actually good news.

We were making a normal GET / request against an OTLP receiver, so we didn't expect a web page there. The important part was that the TCP connection succeeded.

Once the instrumented applications started sending actual OTLP requests, telemetry began appearing in SigNoz.

This debugging experience reinforced something important: observability infrastructure itself needs to be debugged systematically.

Creating a Real Distributed Trace

Once telemetry started flowing, we tested the complete application.

A normal order request follows approximately this path:

POST /orders
      │
      ├── GET /validate
      │      └── Auth Service
      │
      ├── GET /products/{id}
      │      └── Product Service
      │
      └── POST /pay
             └── Payment Service

SigNoz allowed us to see this entire request as a distributed trace rather than four disconnected API calls.

That immediately made the relationship between services much easier to understand.

Intentionally Breaking the Application

A monitoring demo isn't very interesting if everything stays healthy.

So we designed the Payment Service with multiple simulation modes:

normal
slow
fail
timeout
random

A normal payment returns immediately.

The slow mode intentionally waits:

time.sleep(5)

The fail mode generates an HTTP 500 error.

The most interesting scenario was timeout.

The Payment Service waits for 15 seconds, while the Order Service only waits 10 seconds for the response.

That creates a realistic downstream timeout.

From the user's perspective, the order fails.

From an observability perspective, however, something much more interesting happens.

Following a Timeout Through SigNoz

When we triggered the timeout scenario, SigNoz showed the Order Service request taking roughly 10 seconds before returning a 504.

The trace made the cause immediately visible.

Order Service
POST /orders
      │
      └── Payment Service
          POST /pay
          ~15 seconds

Instead of simply knowing that POST /orders failed, we could see that the Order Service was waiting for the Payment Service.

The trace contained the HTTP spans and exception information surrounding the timeout.

We also tested explicit payment failures.

Those produced traces where the Order Service returned an error while the Payment Service showed the downstream failure.

This was where distributed tracing became particularly valuable to us. The error appeared at one API boundary, but the actual cause originated deeper inside the request chain.

Correlating Logs with Traces

Traces tell us where a request spent its time.

Logs help explain what the application was doing.

We added structured application logging to the Payment Service.

For example:

Payment request received
Payment successful
Slow payment detected
Payment failed
Simulating payment timeout

These logs were exported into SigNoz as well.

When testing different scenarios, we could see entries such as:

Payment request received | mode=normal
Payment successful

Payment request received | mode=slow
Slow payment detected

Payment request received | mode=fail
Payment failed

Now we weren't looking at traces and logs as separate debugging systems. They represented different views of the same application behavior.

Metrics and Service Health

OpenTelemetry instrumentation also gave us HTTP metrics.

Inside SigNoz we could inspect metrics such as:

http.server.duration
http.client.duration
http.server.active_requests

The APM view provided an even clearer picture.

During normal requests, latency remained low and error rates stayed around zero.

After triggering slow and timeout scenarios, latency increased sharply and the error percentage changed accordingly.

This allowed us to move from investigating individual requests to understanding the overall health of a service.

Detecting Problems with Alerts

The next step was making the system react automatically.

We experimented with both log-based and trace-based alerts in SigNoz.

For example, a log-based rule can monitor:

service.name = 'payment-service'
AND
severity_text = 'ERROR'

This allows failures from the Payment Service to trigger an alert instead of requiring someone to continuously watch the Logs Explorer.

We also created trace-based conditions for abnormal latency.

For example, if a relevant request crosses a latency threshold, the system can identify it as a critical condition.

This is where the project started moving from observability toward incident response.

Building SentinelOps on Top of SigNoz

We didn't want to rebuild an observability platform.

SigNoz already handles telemetry collection, storage, querying, visualization, tracing, metrics, logs, and alerts.

Instead, SentinelOps adds an automation and incident-management layer on top.

Our Node.js/Express backend communicates with SigNoz and provides normalized data to the SentinelOps frontend.

The rule engine evaluates configured conditions periodically.

Conceptually:

SigNoz telemetry
       │
       ▼
SentinelOps Rule Engine
       │
       ├── Healthy → continue monitoring
       │
       └── Threshold breached
                 │
                 ▼
             Incident
                 │
                 ▼
          Investigation

An incident can contain information such as the affected service, severity, detected condition, timestamp, telemetry evidence, status, and root-cause analysis.

The frontend then allows users to acknowledge, investigate, and resolve those incidents.

Using SigNoz MCP for Root-Cause Investigation

One of the most interesting parts of SentinelOps is the integration with the SigNoz MCP server.

Detecting that something went wrong is only the beginning.

Once an incident exists, we want the system to investigate the telemetry surrounding it.

Instead of manually inspecting several screens, SentinelOps can use SigNoz MCP as the observability interface for an AI-assisted investigation.

The workflow is:

Incident detected
       ↓
Identify service + incident time
       ↓
Query relevant SigNoz telemetry
       ↓
Inspect metrics + traces + logs
       ↓
Correlate evidence
       ↓
Generate root-cause analysis
       ↓
Attach analysis to incident

For our timeout scenario, for example, the investigation should be able to connect the high Order Service latency with the slow downstream Payment Service request and the corresponding timeout/error evidence.

This allows AI to work with real observability context rather than guessing from a single error message.

Making the Environment Reproducible

We also containerized the project using Docker Compose.

Instead of manually launching every service in a different terminal, the complete demo environment can be started together:

docker compose up --build -d

The environment contains our application services, SentinelOps backend, frontend dependencies, and MongoDB, while the instrumented services export telemetry to the configured SigNoz instance.

This makes it significantly easier for another developer—or a hackathon judge—to reproduce the application behavior.

What We Learned

The biggest lesson from building SentinelOps was that collecting telemetry is only the first step.

Metrics are excellent for recognizing that system behavior has changed.

Traces show how a request moved through distributed services.

Logs provide detailed application context.

Alerts help ensure that important conditions aren't missed.

But the real value appears when these signals are connected.

Our Payment Service timeout demonstrated this clearly:

Metric:
Latency increased

Trace:
Order → Payment became slow

Log:
Payment timeout simulation occurred

Alert:
Abnormal condition detected

SentinelOps:
Incident created and investigated

Instead of treating observability as a collection of dashboards, SentinelOps treats telemetry as evidence that can drive an incident-response workflow.

Final Thoughts

Building SentinelOps gave us practical experience with OpenTelemetry instrumentation, distributed tracing, metrics, centralized logging, alerting, Dockerized microservices, and SigNoz.

More importantly, it changed how we thought about observability.

The goal isn't simply to collect more data.

The goal is to make that data useful when something actually breaks.

With SigNoz providing the observability foundation and SentinelOps adding automated detection, incident management, and AI-assisted investigation, we're working toward a workflow where developers can move from:

"Something is broken."

to:

"This service is failing, this downstream operation caused it, and here is the telemetry evidence."

much faster.

That's what we built SentinelOps to do.

DEV Community: Pranjal Verma