When I first started building SitePulse—a cloud monitoring dashboard designed to track server status and health metrics—I ran into a classic developer problem. I could easily ping target servers to see if they were "up," but I had zero visibility into my own backend. When my data ingestion endpoints started lagging, looking at standard terminal output was useless. I didn’t just need to know if an error happened; I needed to know exactly where the bottleneck was in the request lifecycle.
Instead of rewriting my entire logging architecture, I decided to use OpenTelemetry to trace my Python FastAPI backend, sending the data directly to SigNoz. Here is exactly how I set it up in under 30 minutes, and what I learned along the way.
The Goal: From Black Box to Full Tracing
For SitePulse to be a reliable part of my CloudSentinel suite, the FastAPI backend needs to handle high volumes of health-check data rapidly.
My objective was simple:
Automatically instrument all incoming HTTP requests.
Export those traces to a backend where I could visualize latency spikes.
Do it without polluting my business logic with custom logging code.
I chose SigNoz because it natively supports OpenTelemetry (OTel) and gives me both metrics and traces out of the box without the heavy setup of an ELK stack.
Step 1: Setting Up OpenTelemetry in FastAPI
First, I needed to install the core OpenTelemetry libraries and the specific FastAPI auto-instrumentation package.
Bash
pip install opentelemetry-api opentelemetry-sdk
pip install opentelemetry-instrumentation-fastapi
pip install opentelemetry-exporter-otlp
Auto-instrumentation is practically magic. Instead of wrapping every single route in a timer, you attach the OTel instrumentor to the FastAPI app instance. Here is the exact main.py configuration I used for SitePulse:
Python
from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
app = FastAPI(title="SitePulse API")
1. Initialize the Tracer Provider
provider = TracerProvider()
2. Configure the OTLP Exporter to send data to SigNoz
SigNoz typically listens for OTLP gRPC on port 4317
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
3. Add the BatchSpanProcessor (batches spans for better performance)
processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
4. Instrument the FastAPI app
FastAPIInstrumentor.instrument_app(app)
@app.get("/api/v1/health")
async def check_health():
# A simple endpoint to test tracing
return {"status": "operational", "service": "SitePulse"}
Step 2: Routing to SigNoz
For local development, I spun up SigNoz using their official Docker Compose script. The crucial part here is the endpoint="http://localhost:4317" in my Python code. SigNoz’s OTel Collector listens on port 4317 for gRPC traffic.
Once I started my FastAPI server using Uvicorn (uvicorn main:app --reload), I hit the /api/v1/health endpoint a few times and opened the SigNoz dashboard at http://localhost:3301.
Step 3: Finding the Bottleneck
The real value clicked when I looked at a more complex endpoint—the one SitePulse uses to aggregate server metrics.
By default, the FastAPIInstrumentor captures the overall request time. But because OpenTelemetry automatically propagates context, I could also see how much time was spent on database queries within that single request.
Looking at the flame graph in SigNoz, it became instantly obvious that an asynchronous database call was blocking the event loop. I didn't have to guess; the trace visually mapped out exactly where the 400ms delay was happening.
What I Learned (The Gotchas)
If you are doing this for your own project, keep these two things in mind:
gRPC vs HTTP: Make sure you install opentelemetry-exporter-otlp. There are separate packages for HTTP and gRPC. SigNoz defaults to gRPC on 4317. If you accidentally use the HTTP exporter without changing the port/endpoint, your traces will silently fail to export.
Auto-instrumentation is a baseline, not the finish line: FastAPIInstrumentor is great for HTTP spans, but if you have complex background tasks or custom functions, you'll still want to create manual spans using tracer.start_as_current_span("my_custom_logic").
Conclusion
Adding observability to SitePulse didn't require a massive architectural rewrite. With about 15 lines of OpenTelemetry configuration, I went from guessing about API latency to having a real-time, visual breakdown of every request in SigNoz. If you are building a modern web app, skip the print() statements and instrument your code properly from day one.
Top comments (0)