Monitor a FastAPI App with Prometheus + Grafana (Metrics, Dashboards, Alerts)

Your FastAPI app is slow, and you cannot see it. There is no graph of your traffic, no counter of your errors, and nothing that tells you which endpoint is the slow one. The first symptom is a user complaint — and by then you are debugging blind.

In this tutorial you will fix that, end to end, on your own machine: instrument a FastAPI app so it publishes its own metrics, collect them with Prometheus, and build a Grafana dashboard that catches a slow endpoint live — then set an alert that fires when it happens again. Everything is local and free: no accounts, no API keys, no cloud.

What we are using (pinned, so your install matches this post exactly): prometheus-fastapi-instrumentator 8.1.0 · Prometheus 3.16.0 · Grafana OSS 13.3.0 · FastAPI 0.162.4 · uvicorn 0.54.0 · Python 3.12. The complete companion code — the app, the Prometheus config, the load script, the exported dashboard JSON and the alert rule — is in the companion repository, folder topic-10-prometheus-grafana/.

Step 1 — Install the stack

Create a virtual environment and install the app stack. The instrumentator is the only monitoring-specific package the app needs:

$ python -m venv venv && source venv/bin/activate
$ pip install fastapi uvicorn prometheus-fastapi-instrumentator

Pin what you installed in a requirements.txt (fastapi==0.162.4, uvicorn==0.54.0, prometheus-fastapi-instrumentator==8.1.0, and httpx==0.28.1, which the load script in Step 9 uses). Prometheus and Grafana themselves are single binaries you download from their official sites and run locally — no Docker required, which is how this post was produced.

Step 2 — The demo app: one endpoint per signal

The demo app is deliberately small, with one endpoint per thing a dashboard should show you:

import asyncio

from fastapi import FastAPI
from fastapi.responses import JSONResponse

app = FastAPI(title="Demo Shop API")

@app.get("/")
async def home():  # instant JSON — the healthy traffic
    return {"status": "ok", "service": "demo-shop"}

@app.get("/slow")
async def slow():  # sleeps 0.8s, then answers
    await asyncio.sleep(0.8)
    return {"status": "ok", "detail": "that took a while"}

@app.get("/boom")
async def boom():  # HTTP 500, every time
    return JSONResponse(status_code=500, content={"detail": "something broke"})

That file is complete and runnable as written — it is the companion repo's app.py minus the two monitoring lines, which are Step 3.

Run it with uvicorn app:app --port 8000 and curl all three: / answers instantly, /slow takes 0.80s on the timer, and /boom returns a clean HTTP 500. Nothing is monitored yet — that is the point. We are about to watch all three behaviours appear on a dashboard.

Step 3 — Instrument the app (two calls)

This is the whole integration. Add the import and one line after you create the app:

from prometheus_fastapi_instrumentator import Instrumentator

Instrumentator().instrument(app).expose(app)

instrument(app) wraps your app so every request is measured; expose(app) serves those measurements at a new endpoint, /metrics. Restart uvicorn, generate a few requests, and open http://localhost:8000/metrics in your browser.

Step 4 — Read what /metrics gives you for free

Without writing a single metric yourself, the instrumentator publishes five defaults. The three that matter most:

• http_requests_total — a counter of every request, labelled by handler (your endpoint), status and method.
• http_request_duration_seconds — a histogram of how long requests take, in buckets. This is where response-time percentiles come from.
• http_request_size_bytes and http_response_size_bytes — request and response sizes.

One gotcha to learn now, because it bites everyone: the status label is grouped. After hitting /boom, the counter reads http_requests_total{handler="/boom", status="5xx"} — there is no status="500" series. Remember that when we write queries in Step 6, and again in the Traps section at the end.

The duration histogram's default buckets are le = 0.1, 0.5, 1.0, +Inf seconds. A percentile computed from a histogram is an interpolated estimate inside a bucket, not an exact measurement — keep that in mind when we read "0.975 seconds" in Step 9: it is the honest estimate the buckets support, and widening the buckets is how you trade precision for cost.

Step 5 — Point Prometheus at the app

Prometheus pulls metrics: every few seconds it fetches /metrics and stores the numbers as time series. Create prometheus.yml:

global:
  scrape_interval: 5s   # a demo choice — the default is 1m

scrape_configs:
  - job_name: fastapi
    static_configs:
      - targets: ["localhost:8000"]

The 5-second scrape interval is a deliberate demo choice so the dashboard moves while you watch; in production, 15–60 seconds is normal. Start Prometheus with this config (./prometheus --config.file=prometheus.yml from the folder you extracted it to), then open http://localhost:9090/targets: the fastapi target must read UP. If it reads DOWN, stop and fix that first — every later step depends on it (see Traps).

Step 6 — Ask three questions with PromQL

Monitoring dashboards are built from three questions — the RED method: Rate (how much traffic am I getting?), Errors (how much of it is failing?), Duration (how slow is it?). Each one is a single Prometheus query. Try them in Prometheus's own graph tab first (http://localhost:9090/graph).

Rate — requests per second, per endpoint. rate(...) turns the ever-growing counter into a per-second rate over a window — here, the last 5 minutes:

sum by (handler) (rate(http_requests_total{handler!="/metrics", handler!="none"}[5m]))

Errors — the share of responses that are 5xx. Note the grouped label from Step 4: the numerator selects status="5xx", because status="500" does not exist:

sum(rate(http_requests_total{status="5xx"}[5m]))
/
sum(rate(http_requests_total{handler!="/metrics"}[5m]))

Duration — the 95th percentile response time, per endpoint. Read it as: "95% of requests were faster than this." histogram_quantile does the bucket interpolation for you:

histogram_quantile(0.95,
  sum by (handler, le) (rate(http_request_duration_seconds_bucket{handler!="/metrics", handler!="none"}[5m])))

Step 7 — Connect Grafana to Prometheus

Start Grafana (run the grafana server binary from the folder you extracted it to) and sign in at http://localhost:3000 (the first-run login is admin / admin). Go to Connections → Data sources → Add data source → Prometheus, and set the URL to http://localhost:9090:

Click Save & test. You want the green confirmation that Grafana successfully queried the Prometheus API — that message is the whole step done:

Running Grafana in Docker instead? Then localhost is the container, not your machine — use host.docker.internal in the URL. This is the single most common reason a first data source fails its test (see Traps).

Step 8 — Build the RED dashboard

Create a dashboard with three panels — one per question from Step 6, using the same three queries:

• Rate — requests/sec by handler (the rate query)
• Errors — share of 5xx responses (the errors query; set the unit to percent 0–1)
• Duration — p95 and p50 by handler (the duration query, plus the same query at 0.50)

Set the dashboard to refresh every 5 seconds and show the last 15 minutes. Send some gentle traffic to the app and you should see the baseline: about a request a second, a small 5xx share from /boom, and the Duration panel already telling a story — /slow sits up near a second while everything else hugs the floor.

Step 9 — The burst: watch it catch a slow endpoint live

Now attack your own app. The companion repository's load_burst.py runs a four-phase load timeline — baseline, an all-endpoints burst, a slow-heavy phase, then an error-heavy phase. The full run made 2,285 requests: 1,767 to /, 344 to /slow, and 174 to /boom (every one an HTTP 500).

What the dashboard showed, in order:

• Rate climbed first — peaking at roughly 19 requests per second measured on a 2-minute window. (Your dashboard's 5-minute window will show a lower peak, around 6 req/s. Both are correct — a rate is meaningless without its window, which is why this post states the window every time.)
• Errors followed — at the peak of the error-heavy phase, over 11% of responses were failing (11.4% measured).
• Duration is the tell. /slow's p95 read 0.975 seconds — and it had read that since the very first scrape, long before the burst, because the endpoint sleeps 0.8 seconds and the histogram interpolates inside its 0.5–1.0 bucket. The dashboard did not just catch the burst; it had already fingered the slow endpoint at baseline. (The overall p95 across all endpoints peaked at about 0.90 seconds during the burst — remember that number when we set the alert threshold in Step 10.)

Step 10 — One alert: know when it happens again

A dashboard you must watch is a chore; an alert watches for you. In Grafana: Alerting → Alert rules → New alert rule. Query the overall p95 (the duration query without the per-handler split), and set the condition: when the last value of that query is above 0.5, sustained for 1 minute:

Run the burst again and wait. Once the p95 stays above half a second for a full minute, the rule flips to Firing:

Two honest notes. First, this demo wires no notification channel — the rule fires in Grafana's UI, and connecting email, Slack or a webhook (a "contact point") is your next step; this post does not claim a message was delivered. Second, the same rule exists as a Prometheus alerting rule in YAML in the companion repository, for setups that alert from Prometheus instead of Grafana.

Traps (each one cost somebody an hour)

• Target DOWN. If Prometheus shows your target as DOWN, nothing else can work. Usually the app is not running, or the port in prometheus.yml is wrong. Check Targets before debugging queries.
• Docker networking. If Grafana runs in Docker and Prometheus (or the app) runs on your host, localhost inside the data-source URL points at the container. Use host.docker.internal.
• Querying status="500". It returns an empty result — not an error, just nothing — because the instrumentator groups statuses (5xx). An empty graph here almost always means a label value that does not exist, not "no errors".

Recap

You now have the full loop: the app publishes its own numbers at /metrics (two calls to add), Prometheus scrapes them every few seconds, three PromQL queries answer Rate, Errors and Duration, a Grafana dashboard shows them live, and one alert fires when the p95 stays too high. The pattern transfers unchanged to a real application: keep the three questions, watch the labels (grouped statuses!), and always quote a rate with its window.

Practice task: add the instrumentator to an app of your own, find your slowest endpoint's p95 at baseline, and write the number down. If it surprises you, the dashboard has already paid for itself.

FAQ

Do I need Docker for Prometheus and Grafana?
No. Both are single binaries that run locally, which is how this tutorial was produced. Docker works too — just mind the host.docker.internal note in Step 7.

Why is my p95 an estimate?
Because it comes from a histogram: Prometheus knows how many requests fell into each duration bucket, and histogram_quantile interpolates inside the bucket. It is the right tool for spotting slow endpoints, not a stopwatch.

What does this cost to run in production?
Prometheus and Grafana OSS are free and self-hosted. The real cost is storage, which you control with the scrape interval and retention settings — start at 15–60 second scrapes, not the 5 seconds this demo used.

Sources

• prometheus-fastapi-instrumentator — PyPI project page (the instrumentator's defaults and label behaviour)
• Prometheus — Querying basics (rate, histogram_quantile, label matchers)
• Grafana — Alerting documentation (alert rules, conditions, pending/firing states)
• All measurements in this post (request counts, rates, percentiles, the firing alert) are from the author's own run of the companion code, November 2026.

This post was originally published on Visual Studio Learn, where the video version and the free companion code are linked.