How I Tracked Down a Database Latency Anomaly in a FastAPI App (Using SigNoz & OTel)

A few days ago, I was looking into setting up proper tracing for a local FastAPI backend. While local unit tests run in milliseconds, database latency has a habit of creeping up silently when you start querying tables in real-world environments. I wanted to see if I could use OpenTelemetry (OTel) and SigNoz to capture trace graphs and pinpoint performance issues.

In this write-up, I'll walk through exactly how I set up the environment, instrumented the Python code, ran into a frustrating Windows Docker loopback issue, resolved it, and eventually captured the exact database bottleneck in a trace waterfall chart.

The Setup: SigNoz on Windows
For the backend telemetry store, I went with SigNoz. It's OpenTelemetry-native, meaning you don't need any vendor-specific helper libraries in your code. You just push standard OTel data, and it visualizes it.

Since I'm running Windows, I installed Docker Desktop and set up WSL2. The hackathon rules specify using Foundry (SigNoz's CLI manager) for deployment.

I created a basic casting.yaml file:

apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
  mcp:
    spec:
      enabled: true

And deployed it via the CLI:
.\foundryctl.exe cast -f casting.yaml

Foundry generated the configuration and spun up the containers (ClickHouse, PostgreSQL, Query Service, and the OTel Collector). The dashboard interface was up immediately on http://localhost:8080.

Code & Instrumentation: The "Gotcha" on Windows
Next, I wrote a simple FastAPI application with a SQLite database backend using SQLAlchemy. I set up three basic endpoints:

/checkout - A normal database insert.
/slow-db - An endpoint simulating a slow database transaction using a manual sleep.
/error - An endpoint that triggers an HTTP 500 error.
The Connection Refused Anomaly
When I first ran the app and pointed the OpenTelemetry gRPC exporter to http://localhost:4317 (the default port), the terminal filled up with connection resets: UNAVAILABLE: ipv4:127.0.0.1:4317: End of TCP stream

It turned out to be a double issue:

WSL Port Forwarding: On Windows, Docker Desktop handles loopback routing through wslrelay.exe. Sometimes, HTTP/2 gRPC channels multiplexing over loopback interfaces get closed immediately by the relay tunnel.
Collector Config: The startup config pulled by the collector from SigNoz set all active pipelines to No-Op (nop).
To fix this:

I modified the generated compose.yaml (inside ./pours/deployment/) to remove the --manager-config flag from the ingester container, forcing it to stick to our local configuration.
I switched the Python app's exporter from OTLP/gRPC (port 4317) to OTLP/HTTP (port 4318) and pointed it to 127.0.0.1 directly.
This resolved the connection drops, and the telemetry began exporting cleanly.

Here is the final main.py code:

import time
from fastapi import FastAPI, HTTPException
from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor

# Configure the tracer provider and set the service name
resource = Resource(attributes={"service.name": "demo-fastapi-service"})
provider = TracerProvider(resource=resource)

# Export traces locally using OTLP/HTTP to the local collector on port 4318
processor = BatchSpanProcessor(
    OTLPSpanExporter(endpoint="http://127.0.0.1:4318/v1/traces")
)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

app = FastAPI(title="SigNoz Demo Application")

# SQLite Database Setup
DATABASE_URL = "sqlite:///./test.db"
engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False})
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base = declarative_base()

class Item(Base):
    __tablename__ = "items"
    id = Column(Integer, primary_key=True, index=True)
    name = Column(String, index=True)
    price = Column(Integer)

Base.metadata.create_all(bind=engine)

# Auto-instrument all SQLAlchemy queries
SQLAlchemyInstrumentor().instrument(engine=engine)

@app.get("/")
def read_root():
    return {"message": "Welcome to the SigNoz demo application!"}

@app.get("/checkout")
def checkout():
    db = SessionLocal()
    db_item = Item(name="Standard Widget", price=100)
    db.add(db_item)
    db.commit()
    db.refresh(db_item)
    db.close()
    return {"status": "success", "item_id": db_item.id}

@app.get("/slow-db")
def slow_db():
    db = SessionLocal()
    # Manual span to isolate database transaction latency
    with tracer.start_as_current_span("slow_database_transaction"):
        time.sleep(1.5)  # Simulate slow query or locking behavior
        items = db.query(Item).all()
    db.close()
    return {"status": "slow_success", "items_count": len(items)}

@app.get("/error")
def trigger_error():
    raise HTTPException(status_code=500, detail="Simulated internal server error!")

# Auto-instrument incoming HTTP requests
FastAPIInstrumentor.instrument_app(app)

Simulating Traffic
I ran a quick Python loop in the background to send 100 requests spread randomly across /, /checkout, /slow-db, and /error.

import urllib.request
import time
import random

base_url = "http://127.0.0.1:8000"
endpoints = ["/", "/checkout", "/slow-db", "/error"]

for i in range(100):
    selected = random.choice(endpoints)
    try:
        urllib.request.urlopen(base_url + selected)
    except Exception:
        pass
    time.sleep(random.uniform(0.1, 0.5))

This quickly populated our dashboard charts.

What the Traces Showed
After signing into the dashboard, I could see demo-fastapi-service listed under Services. Average latency for /slow-db was hovering around 1.5 seconds, while /checkout was running in less than 5 milliseconds.

I drilled into the traces for /slow-db to view a trace waterfall:

The trace waterfall instantly exposed the latency profile:

The entire HTTP request took 1.51 seconds.
The custom span slow_database_transaction accounted for 1.50 seconds of that time.
The actual database select query (SELECT items.id...) completed in just 0.3 milliseconds.
This is why distributed tracing is so valuable. If you only look at SQL logs, the database looks healthy because the query executed in microseconds. But by wrapping the transaction logic in a span, we immediately see that the application thread was blocked for 1.5 seconds inside the transaction wrapper.

I also checked the traces for /error:

FastAPIInstrumentor automatically caught the HTTP 500 exception and attached the full Python traceback directly to the span attributes, meaning I didn't need to search through stdout log streams to find the crash details.

Wrap-up
Getting a local OTel environment running on Windows took a bit of debugging around Docker loopback ports, but once the OTLP/HTTP traffic was flowing, tracing performance bottlenecks became incredibly visual. The fact that the entire instrumentation takes about 15 lines of python code—without depending on any proprietary vendor SDKs—shows how powerful OpenTelemetry has become.