DEV Community

Bandari Vishnu
Bandari Vishnu

Posted on

I Thought My Python API Was Fast—Until SigNoz Showed Me Where the Time Went

#signoz #opentelemetry #observability #python

When an application works on my machine, it is easy to assume everything is fine.

The API returns a response. The terminal shows no obvious problem. The application is running.

But that does not answer some important questions:

  • Which endpoint is slow?
  • How many requests are failing?
  • What caused an HTTP 500 error?
  • How long did a request actually take?
  • Can I detect failures before manually checking the application?
  • Can AI help explain an error and suggest a possible fix?

I wanted to explore these questions by building a small AI-powered observability project using Python, Flask, OpenTelemetry, SigNoz, and Gemini AI.

What started as a simple Python API became a hands-on experiment in tracing, dashboards, alerts, error analysis, and AI-assisted troubleshooting.

What I Built

The project is a Python Flask application with intentionally different application behaviors:

  • /fast — returns a fast response
  • /slow — intentionally waits before responding
  • /error — intentionally generates an HTTP 500 error
  • /health — checks application health
  • /ai-analysis — analyzes an error and returns a possible issue, root cause, severity, and suggested fix

The application is instrumented with OpenTelemetry and sends telemetry data to SigNoz.

I then created a custom observability dashboard to monitor:

  • Total Requests
  • HTTP 500 Errors
  • Request Latency
  • Requests by Endpoint

I also added an HTTP 500 alert and an AI-powered error analysis component.

Project Architecture

The overall flow of the project is:

User / Test Requests
        |
        v
Python Flask Application
        |
        v
OpenTelemetry Instrumentation
        |
        v
      SigNoz
   /     |      \
Traces Dashboard Alerts

Application Error
        |
        v
AI Error Analyzer
   /           \
Gemini AI   Rule-Based Fallback
Enter fullscreen mode Exit fullscreen mode

This gave me two different layers of troubleshooting:

  1. SigNoz tells me what happened in the application.
  2. The AI analyzer helps explain what the error might mean and what I can investigate next.

Step 1: Creating the Python API

I created a simple Flask application with multiple endpoints.

The fast endpoint returns immediately:

@app.route("/fast")
def fast():
    logger.info("Fast endpoint called")
    return jsonify({
        "endpoint": "fast",
        "status": "healthy",
        "message": "This response was fast!"
    })
Enter fullscreen mode Exit fullscreen mode

The slow endpoint intentionally adds latency:

@app.route("/slow")
def slow():
    logger.warning("Slow endpoint called - simulating latency")
    time.sleep(2)

    return jsonify({
        "endpoint": "slow",
        "status": "slow",
        "duration": "2 seconds",
        "message": "This response was intentionally delayed by 2 seconds"
    })
Enter fullscreen mode Exit fullscreen mode

The error endpoint intentionally generates a failure:

@app.route("/error")
def error():
    logger.error("Error endpoint called")
    raise RuntimeError(
        "Intentional demo error for SigNoz observability"
    )
Enter fullscreen mode Exit fullscreen mode

These endpoints gave me predictable traffic patterns that I could observe inside SigNoz.

Step 2: Instrumenting the Application with OpenTelemetry

The application was started using OpenTelemetry auto-instrumentation:

OTEL_SERVICE_NAME=python-signoz-demo \
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
OTEL_EXPORTER_OTLP_PROTOCOL=grpc \
opentelemetry-instrument python app.py
Enter fullscreen mode Exit fullscreen mode

The service name was set to:

python-signoz-demo
Enter fullscreen mode Exit fullscreen mode

Once the application started sending telemetry, I generated traffic using requests to the different endpoints:

curl http://localhost:5000/fast
curl http://localhost:5000/slow
curl http://localhost:5000/error
Enter fullscreen mode Exit fullscreen mode

I also generated repeated requests so that the dashboard had enough data to visualize:

for i in {1..10}; do
  curl -s http://localhost:5000/fast > /dev/null
  curl -s http://localhost:5000/slow > /dev/null
done
Enter fullscreen mode Exit fullscreen mode

For HTTP 500 errors:

for i in {1..20}; do
  curl -s http://localhost:5000/error > /dev/null
  sleep 1
done
Enter fullscreen mode Exit fullscreen mode

This is where the project became much more interesting.

Instead of only seeing a terminal response, I could now inspect the behavior of the application through telemetry.

Step 3: Building the SigNoz Dashboard

I created a custom dashboard called:

Python AI Observability Dashboard

The dashboard included four main panels.

Total Requests

This panel shows the total number of requests received by the application.

It gives a quick overview of application traffic.

HTTP 500 Errors

This panel tracks failed requests.

Because the /error endpoint intentionally raises a RuntimeError, I could immediately see the failures appearing in the dashboard.

Request Latency

This was one of the most useful panels.

The /fast endpoint responds almost immediately, while the /slow endpoint intentionally waits for approximately two seconds.

Seeing the difference visually made the value of observability very clear.

Without telemetry, I only knew that a request "felt slow."

With tracing and latency data, I could see where the time went.

Requests by Endpoint

This panel helped compare traffic across:

/
/fast
/slow
/error
Enter fullscreen mode Exit fullscreen mode

This made it easier to understand which routes were receiving traffic and which endpoints were responsible for failures or increased latency.

Step 4: Investigating HTTP 500 Errors

The /error endpoint intentionally generates:

RuntimeError: Intentional demo error for SigNoz observability
Enter fullscreen mode Exit fullscreen mode

Instead of only seeing a generic response such as:

500 Internal Server Error
Enter fullscreen mode Exit fullscreen mode

observability data can help provide more context around the failed request.

This is one of the biggest lessons I learned from the project:

A status code tells you that something failed. Observability helps you investigate what happened around that failure.

Tracing makes it possible to inspect individual requests, their duration, status, attributes, and associated errors.

Step 5: Creating an HTTP 500 Alert

Dashboards are useful when someone is actively watching them.

But real systems also need proactive detection.

I created an alert for HTTP 500 errors in the Python service.

The goal was simple:

If the application starts generating server errors, the observability system should detect the condition instead of waiting for someone to manually discover it.

This adds another important layer to the project:

Application Failure
        ↓
Telemetry
        ↓
SigNoz
        ↓
Alert Detection
Enter fullscreen mode Exit fullscreen mode

The combination of dashboards and alerts provides both:

  • Visual investigation
  • Proactive failure detection

Step 6: Adding AI-Powered Error Analysis

After building the observability layer, I wanted to experiment with another idea:

Can AI help turn a raw error message into a more understandable troubleshooting summary?

I created an /ai-analysis endpoint.

For example:

curl -G \
  --data-urlencode "error=Database connection timeout" \
  http://localhost:5000/ai-analysis
Enter fullscreen mode Exit fullscreen mode

The analyzer returns structured information such as:

{
  "severity": "HIGH",
  "issue": "Database connection establishment timed out",
  "root_cause": "Potential database connection or resource issue",
  "suggested_fix": "Inspect database utilization, connection pools, network connectivity, and long-running queries",
  "analysis_source": "Gemini AI"
}
Enter fullscreen mode Exit fullscreen mode

The purpose of the AI component is not to replace observability.

Instead, the idea is:

Telemetry → Evidence
AI → Interpretation Assistance
Developer → Final Decision
Enter fullscreen mode Exit fullscreen mode

SigNoz provides the telemetry and evidence.

The AI analyzer attempts to explain the error in a more structured format and suggest areas to investigate.

Step 7: Adding a Rule-Based Fallback

While testing the Gemini integration, I encountered real API availability problems.

At one point, a model was unavailable. At another point, the API returned a temporary high-demand error.

Instead of allowing the entire analysis feature to fail, I added a rule-based fallback.

For example, if an error contains the word timeout, the fallback can return:

{
  "severity": "MEDIUM",
  "issue": "Request timeout detected",
  "root_cause": "The operation took longer than expected.",
  "suggested_fix": "Check slow dependencies, database queries, and external APIs.",
  "analysis_source": "rule-based fallback"
}
Enter fullscreen mode Exit fullscreen mode

This taught me another useful engineering lesson:

An AI-powered feature should not necessarily make the entire application dependent on the availability of an external AI service.

The final design therefore supports:

Error
  |
  v
Try Gemini AI
  |
  +---- Success ----> AI Analysis
  |
  +---- Failure ----> Rule-Based Fallback
Enter fullscreen mode Exit fullscreen mode

This made the project more resilient.

A Real Debugging Problem I Faced

One of the issues I encountered was:

Address already in use
Port 5000 is in use by another program.
Enter fullscreen mode Exit fullscreen mode

I checked which process was using the port:

sudo lsof -i :5000
Enter fullscreen mode Exit fullscreen mode

Then I stopped the old process before restarting the instrumented application.

This was a simple issue, but it reinforced an important point: observability projects involve more than creating dashboards. You also need to understand the application process, ports, telemetry pipeline, instrumentation, and the environment in which everything is running.

Another Challenge: Querying the Correct Latency Field

While creating the request latency panel, I initially tried to query a field that was not available.

The dashboard returned:

field `duration` not found
Enter fullscreen mode Exit fullscreen mode

After checking the available trace fields, I used the correct duration field for the telemetry data.

That small debugging step was valuable because it forced me to understand the actual structure of the collected trace data instead of assuming field names.

Final Testing

I tested the complete application using:

curl http://localhost:5000/health
curl http://localhost:5000/fast
curl http://localhost:5000/slow
curl http://localhost:5000/error
Enter fullscreen mode Exit fullscreen mode

And for dynamic AI analysis:

curl -G \
  --data-urlencode "error=Database connection timeout" \
  http://localhost:5000/ai-analysis
Enter fullscreen mode Exit fullscreen mode

The application successfully demonstrated:

  • Healthy requests
  • Fast requests
  • Slow requests
  • HTTP 500 errors
  • OpenTelemetry instrumentation
  • SigNoz observability
  • Custom dashboards
  • Error alerting
  • AI-assisted error analysis
  • Rule-based fallback analysis

Project Structure

The main project files are:

python-ai-observability-signoz/
├── .gitignore
├── README.md
├── ai_analyzer.py
├── app.py
└── requirements.txt
Enter fullscreen mode Exit fullscreen mode

app.py contains the Flask application and endpoints.

ai_analyzer.py contains the AI-assisted error analysis logic and fallback mechanism.

requirements.txt contains the Python dependencies required to reproduce the project.

What I Learned

This project changed how I think about application monitoring.

1. "The application is running" is not enough

A service can be running while some endpoints are slow or failing.

2. HTTP 500 is only the beginning of the investigation

Knowing that a request failed is useful, but traces and error details provide the context needed for troubleshooting.

3. Latency becomes easier to understand when it is visualized

The difference between the /fast and /slow endpoints became immediately visible in the dashboard.

4. Alerts reduce dependence on manual monitoring

Instead of continuously watching a dashboard, alerts can detect important failure conditions.

5. AI can assist troubleshooting, but telemetry should remain the source of truth

AI-generated explanations can be useful, but they should be grounded in actual application errors and observability data.

6. Fallbacks matter

External AI services can be unavailable, overloaded, or changed. A fallback strategy makes the system more resilient.

The Bigger Idea

The most interesting part of this project was combining traditional observability with AI-assisted analysis.

A possible future workflow could look like this:

Application
    ↓
OpenTelemetry
    ↓
SigNoz
    ↓
Anomaly or Error Detected
    ↓
Relevant Telemetry Collected
    ↓
AI-Assisted Analysis
    ↓
Suggested Root Cause and Next Investigation Steps
    ↓
Developer Review
Enter fullscreen mode Exit fullscreen mode

There is still a lot that could be improved.

Future enhancements could include:

  • Automatically retrieving real trace context for AI analysis
  • Sending alerts to Slack or another notification channel
  • Correlating logs and traces
  • Adding database telemetry
  • Detecting unusual latency patterns
  • Generating AI summaries from actual telemetry
  • Adding authentication and production deployment
  • Containerizing the complete application

Final Thoughts

I started this project with a simple Python API.

By the end, I had explored:

  • Flask application monitoring
  • OpenTelemetry instrumentation
  • Distributed tracing
  • Request latency
  • HTTP 500 error monitoring
  • Custom dashboards
  • Alerting
  • AI-assisted root cause analysis
  • Fallback mechanisms

The biggest lesson for me was simple:

You cannot improve what you cannot see.

And sometimes, an API that looks fast from the outside has a very different story inside its traces.

Source Code

The complete project is available on my GitHub repository:

Python AI Observability with SigNoz

Technologies Used

  • Python
  • Flask
  • OpenTelemetry
  • SigNoz
  • Gemini AI
  • Docker
  • Git
  • GitHub

If you found this project interesting, I would love to hear your feedback and ideas for improving the AI-assisted observability workflow.

Top comments (0)