DEV Community

Jordan
Jordan

Posted on

Monitoring My Flask App with Prometheus and Grafana

#devops #monitoring #docker #python

Introduction

For an earlier IT 3200 assignment, I built a small Flask app and ran it in Docker on a Linux virtual machine. The app displays a simple HTML page. For this assignment, I added monitoring so I could see how many requests it received, whether any requests failed, and how long the app took to respond.

I used Google Compute Engine to host the app, Prometheus to collect metrics, and Grafana to display them. I tested the setup by sending requests to the app and checking whether the graphs changed.

Everything runs on one Linux VM called vm-it3200-m7. It has three Docker containers: student-app, prometheus, and grafana. This was a small class project, so I kept the setup on one server.

Why Prometheus and Grafana?

I chose Prometheus because I had already used it in the Cloud Watching assignment. It checks a metrics endpoint at regular intervals and saves the numbers over time. Its query language, PromQL, lets me look at things like request counts and response times. Those were useful measurements for my Flask app.

I added Grafana so I could see several graphs together instead of running each query separately. Prometheus collects and stores the metrics, and Grafana reads that data to build the dashboard.

Google Cloud's monitoring service would have been another option, but Prometheus let me build on the work I had already done. Loki or the Elastic Stack would make more sense if I wanted to search through logs. For this project, I focused on request counts, errors, and response times.

There are limits to this setup. Checking every 15 seconds can miss short problems between checks, and the app's counters reset when its process restarts. I also did not add tracing or a central log system. Since everything is on one VM, a VM failure would take down both the app and its monitoring tools.

Architecture

Architecture Diagram

The firewall rule allows my public IPv4 address using a /32 range and applies to the VM through the it3200-m7 network tag. I used port 3000 for Flask, 9090 for Prometheus, and 3001 for Grafana. Grafana uses port 3000 inside its container, so Docker maps the VM's port 3001 to the container's port 3000.

The containers communicate through a Docker network called it3200-monitoring. Prometheus connects to http://student-app:3000/metrics, and Grafana connects to http://prometheus:9090. Docker lets them find each other by container name, so they do not need to use the VM's public IP to communicate. Someone with administrator access to Docker can still access these containers.

Prometheus saves its data in a Docker volume named prometheus-data and keeps it for two days. Grafana saves its settings in grafana-data. Replacing a container does not erase these volumes, but deleting the VM's storage would. That is why I saved my screenshots and configuration on my computer.

Installation and configuration

First, I installed Docker on the Ubuntu VM:

sudo apt-get update
sudo apt-get install -y docker.io curl
sudo systemctl enable --now docker
Enter fullscreen mode Exit fullscreen mode

My app already had a Dockerfile using Python 3.11. It installs the packages in requirements.txt, copies the app files, and runs python app.py on port 3000. I reused that Dockerfile.

I added this dependency to requirements.txt:

prometheus-flask-exporter==0.23.2
Enter fullscreen mode Exit fullscreen mode

Then I added the exporter right after creating the Flask app:

from flask import Flask, render_template
from prometheus_flask_exporter import PrometheusMetrics

app = Flask(__name__)
metrics = PrometheusMetrics(app, group_by='endpoint')
Enter fullscreen mode Exit fullscreen mode

This adds a /metrics page that Prometheus can read. The group_by setting groups request times by the Flask endpoint instead of every possible URL. The exporter also leaves its own metrics requests out of the default request measurements. Otherwise, checking the app could look like extra user traffic.

I configured Prometheus to check two targets every 15 seconds:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: flask-app
    metrics_path: /metrics
    static_configs:
      - targets: ['student-app:3000']
Enter fullscreen mode Exit fullscreen mode

The first target is Prometheus itself, which I used for Cloud Watching. The second is my Flask app. One thing I learned was that seeing Prometheus marked UP does not automatically mean it is collecting data from the app too.

I rebuilt the application and created the Docker network:

sudo docker build -t student-app:monitoring .
sudo docker network create it3200-monitoring
Enter fullscreen mode Exit fullscreen mode

I used observability/deploy.sh to start the containers. The script builds the app and checks the Prometheus configuration with promtool check config. It also stops and renames the previous containers so they are kept as backups. The new containers use the shared Docker network and saved data volumes. Docker logs are limited to three 10 MB files per container to help prevent them from filling the disk.

These are the Prometheus settings for the configuration file, storage, and retention:

-v "$PWD/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro"
-v prometheus-data:/prometheus
--config.file=/etc/prometheus/prometheus.yml
--storage.tsdb.path=/prometheus
--storage.tsdb.retention.time=2d
Enter fullscreen mode Exit fullscreen mode

These lines are part of the Docker command, so they should not be run by themselves. The full script is included at the end of this post. If the containers or network already exist, check their names before running the commands again.

I used this file to tell Grafana where to find Prometheus:

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s
Enter fullscreen mode Exit fullscreen mode

Grafana loads this file from /etc/grafana/provisioning/datasources. Another configuration file loads the dashboard JSON from /var/lib/grafana/dashboards. This saves the dashboard setup in files so I can recreate it without setting up every panel by hand.

I included the full script and monitoring files at the end of this post. Each code block shows where to save the file in the app project.

I used the latest Docker tags for Prometheus and Grafana. The script records the downloaded image IDs and digests in deployed-images.txt. Since latest can change, using those recorded digests would be a better way to repeat the setup with the exact same versions.

What the dashboard measures

The dashboard contains five panels:

Panel Purpose
Flask scrape availability Shows whether Prometheus can collect Flask metrics.
Requests per second Shows the rate of application HTTP requests.
HTTP errors per second Shows 4xx and 5xx response rates.
Mean response duration Shows mean server-side request processing time.
Requests by HTTP status Separates cumulative request counters by status code.

Request rate uses:

sum(rate(flask_http_request_total{job="flask-app"}[1m]))
Enter fullscreen mode Exit fullscreen mode

The request counter keeps increasing as requests come in. rate uses the last minute of samples to estimate requests per second. It can account for a counter resetting, but it needs more than one sample before it can show a rate.

Mean response duration uses the histogram's sum and count:

sum(rate(flask_http_request_duration_seconds_sum{job="flask-app"}[1m]))
/
sum(rate(flask_http_request_duration_seconds_count{job="flask-app"}[1m]))
Enter fullscreen mode Exit fullscreen mode

The duration metric measures how long the server takes to process requests. It does not include the full time a browser waits over the internet. The graph can have gaps when there are no requests. I created dashboards for this project but did not set up automatic alerts.

Showing it working

3 running docker containers

The Docker output shows all three containers running, and the readiness request returns Prometheus Server is Ready.

Prometheus UP query

The up query returned 1 for both Prometheus and Flask, showing that both could be scraped. Grafana is not listed here because it reads data from Prometheus instead of being a scrape target in my setup.

Grafana Dashboard

To test it, I sent 120 requests to the home page with a half-second pause between requests. Then I sent ten requests to /missing-page. That page does not exist, so it returns a 404. This gave me a simple way to check whether errors appeared on the dashboard.

for i in $(seq 1 120); do
  curl -fsS http://localhost:3000/ > /dev/null
  sleep 0.5
done

for i in $(seq 1 10); do
  curl -s -o /dev/null -w '%{http_code}\n' http://localhost:3000/missing-page
  sleep 1
done
Enter fullscreen mode Exit fullscreen mode

In the screenshot, Flask scrape availability stays at 1 across the displayed samples. Around 22:17, the request rate peaks at about 1.9 requests per second and then drops to about 0.4 as the test finishes. The status graph shows about 120 successful responses and ten 404 responses. These numbers are estimates from reading the graph.

The error graph starts at zero and then rises to about 0.11 errors per second during the missing-page test. The errors were expected because I requested a page that did not exist. The displayed rate is averaged over a one-minute window, which is why it is lower than the one request per second sent by the test.

The average response time is roughly 230–260 microseconds, or 0.23–0.26 milliseconds. Grafana displays these small values in microseconds even though the original metric uses seconds. The requests came from inside the VM and the app is simple, so this does not tell me how fast the app would be for users over the internet or under a much larger load. Empty sections of the rate and duration graphs can also mean there were not enough recent requests or samples.

The graphs matched the activity I generated: the request count increased, the missing-page requests showed up as 404s, and Prometheus continued collecting metrics. Browser visits can add requests too, so I would not expect the counters to match a test script exactly every time.

The updated app also passed three local checks: successful requests were counted and timed, missing pages were counted as 404s, and metrics scrapes did not inflate the app request count. I still needed the VM checks and screenshots to confirm that the containers and dashboard worked together.

Roadblocks and lessons learned

My first roadblock was getting the VM running again. I had deleted the previous Azure resource group to avoid costs. When I tried creating another VM, Azure blocked it because of a subscription region policy. I switched to Google Compute Engine and reused the same Dockerized app.

Another thing I learned was the difference between monitoring Prometheus and monitoring my app. Cloud Watching only needed Prometheus to monitor itself. Adding the Flask exporter and a second target let me see what the app was doing. I also learned why the target uses student-app:3000: inside the Prometheus container, localhost points back to Prometheus, not Flask.

Security and cost considerations

I restricted the firewall rule to my public IPv4 address and the ports needed for the lab. Grafana requires an admin password, and new account sign-ups are disabled. I entered the password when running the script instead of saving it in the code. Someone with Docker administrator access could still inspect container settings, so this is not a complete solution for storing production secrets.

This setup uses HTTP, so restricting the source IP does not encrypt the connection. For a longer-term deployment, I would use HTTPS or an SSH tunnel, keep metrics endpoints private, and use separate accounts with limited permissions.

I did not put student IDs, passwords, or request text into metric labels. Grouping by endpoint also keeps the number of labels manageable. I selected screenshots that do not show the student ID on my original app page. The app still uses Flask's development server, which I would need to replace and configure properly before using it in production.

To keep storage use down, Prometheus keeps two days of metrics and Docker rotates the container logs. Prometheus and Grafana are open source, but the Google VM, disk, network traffic, and public IP can still cost money. Once my screenshots and files were saved, I deleted the lab resources.

Conclusion

This project helped me see what monitoring adds to an app that is already running. I could generate requests and see the traffic, errors, and response times change on the dashboard. Prometheus handled collecting the metrics, and Grafana made them easier to compare. If I continued working on this setup, my next steps would be secure access and alerts for availability or high error rates.

Official references

Full setup files

Deployment script and monitoring files

observability/deploy.sh 

#!/usr/bin/env bash
# Run on the Ubuntu VM: bash observability/deploy.sh
set -euo pipefail
cd "$(dirname "$0")/.."
ROOT="$PWD"

# Prepare and validate before replacing any running containers.
sudo docker build -t student-app:monitoring .
sudo docker pull prom/prometheus:latest
sudo docker pull grafana/grafana:latest
sudo docker run --rm --entrypoint /bin/promtool \
  -v "$ROOT/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  prom/prometheus:latest check config /etc/prometheus/prometheus.yml

for name in student-app prometheus grafana; do
  if sudo docker container inspect "${name}-previous" >/dev/null 2>&1; then
    echo "Backup container ${name}-previous already exists. Stop here and review before deploying again."
    exit 1
  fi
done

read -rsp 'Choose a NEW Grafana admin password (at least 12 characters): ' ADMIN_PASSWORD
printf '\n'
if [ "${#ADMIN_PASSWORD}" -lt 12 ]; then
  echo 'Password too short. Nothing has been replaced.'
  exit 1
fi
umask 077
ENV_FILE="$(mktemp)"
trap 'rm -f -- "$ENV_FILE"' EXIT
printf 'GF_SECURITY_ADMIN_PASSWORD=%s\n' "$ADMIN_PASSWORD" > "$ENV_FILE"
unset ADMIN_PASSWORD

sudo docker network inspect it3200-monitoring >/dev/null 2>&1 || sudo docker network create it3200-monitoring
sudo docker volume create prometheus-data >/dev/null
sudo docker volume create grafana-data >/dev/null

# Keep stopped backups. Do not delete the old containers or their volumes.
for name in student-app prometheus grafana; do
  if sudo docker container inspect "$name" >/dev/null 2>&1; then
    sudo docker stop "$name"
    sudo docker rename "$name" "${name}-previous"
  fi
done

sudo docker run -d --name student-app --restart unless-stopped \
  --network it3200-monitoring -p 3000:3000 \
  --log-opt max-size=10m --log-opt max-file=3 student-app:monitoring

sudo docker run -d --name prometheus --restart unless-stopped \
  --network it3200-monitoring -p 9090:9090 \
  --log-opt max-size=10m --log-opt max-file=3 \
  -v "$ROOT/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  -v prometheus-data:/prometheus prom/prometheus:latest \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/prometheus --storage.tsdb.retention.time=2d

sudo docker run -d --name grafana --restart unless-stopped \
  --network it3200-monitoring -p 3001:3000 \
  --log-opt max-size=10m --log-opt max-file=3 \
  --env-file "$ENV_FILE" \
  -e GF_SECURITY_ADMIN_USER=admin -e GF_USERS_ALLOW_SIGN_UP=false \
  -v grafana-data:/var/lib/grafana \
  -v "$ROOT/observability/grafana/provisioning:/etc/grafana/provisioning:ro" \
  -v "$ROOT/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro" \
  grafana/grafana:latest

echo 'Containers started. Verify readiness, targets, and the dashboard before claiming success.'
sudo docker ps
sudo docker image inspect student-app:monitoring prom/prometheus:latest grafana/grafana:latest \
  --format '{{.Id}} {{json .RepoDigests}}' > "$ROOT/observability/deployed-images.txt"
echo 'Image IDs/digests saved in observability/deployed-images.txt.'
Enter fullscreen mode Exit fullscreen mode

observability/prometheus.yml 

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: flask-app
    metrics_path: /metrics
    static_configs:
      - targets: ['student-app:3000']
Enter fullscreen mode Exit fullscreen mode

observability/grafana/provisioning/datasources/prometheus.yml 

apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s
Enter fullscreen mode Exit fullscreen mode

observability/grafana/provisioning/dashboards/default.yml 

apiVersion: 1
providers:
  - name: IT3200
    folder: IT3200
    type: file
    disableDeletion: true
    editable: false
    options:
      path: /var/lib/grafana/dashboards
Enter fullscreen mode Exit fullscreen mode

observability/grafana/dashboards/flask.json 

{
  "uid": "uvu-flask",
  "title": "UVU Flask Application",
  "schemaVersion": 39,
  "version": 1,
  "timezone": "browser",
  "refresh": "15s",
  "time": {
    "from": "now-15m",
    "to": "now"
  },
  "tags": [
    "IT3200"
  ],
  "panels": [
    {"id":1,"title":"Flask scrape availability","type":"timeseries","description":"Successful scrape is 1; failed scrape is 0.","datasource":{"type":"prometheus","uid":"prometheus"},"gridPos":{"x":0,"y":0,"w":12,"h":8},"targets":[{"refId":"A","expr":"up{job=\"flask-app\"}","legendFormat":"{{job}}","range":true}],"fieldConfig":{"defaults":{"unit":"short","min":0},"overrides":[]},"options":{"legend":{"displayMode":"list","placement":"bottom"}}},
    {"id":2,"title":"Requests per second","type":"timeseries","description":"Includes app requests; excludes /metrics scrapes.","datasource":{"type":"prometheus","uid":"prometheus"},"gridPos":{"x":12,"y":0,"w":12,"h":8},"targets":[{"refId":"A","expr":"sum(rate(flask_http_request_total{job=\"flask-app\"}[1m]))","legendFormat":"{{job}}","range":true}],"fieldConfig":{"defaults":{"unit":"reqps","min":0},"overrides":[]},"options":{"legend":{"displayMode":"list","placement":"bottom"}}},
    {"id":3,"title":"HTTP errors per second","type":"timeseries","description":"4xx and 5xx responses. No error series displays zero; check availability separately.","datasource":{"type":"prometheus","uid":"prometheus"},"gridPos":{"x":0,"y":8,"w":12,"h":8},"targets":[{"refId":"A","expr":"sum(rate(flask_http_request_total{job=\"flask-app\",status=~\"4..|5..\"}[1m])) or vector(0)","legendFormat":"{{job}}","range":true}],"fieldConfig":{"defaults":{"unit":"reqps","min":0},"overrides":[]},"options":{"legend":{"displayMode":"list","placement":"bottom"}}},
    {"id":4,"title":"Mean response duration","type":"timeseries","description":"Mean app processing duration, not browser end-to-end latency. No traffic produces a gap.","datasource":{"type":"prometheus","uid":"prometheus"},"gridPos":{"x":12,"y":8,"w":12,"h":8},"targets":[{"refId":"A","expr":"sum(rate(flask_http_request_duration_seconds_sum{job=\"flask-app\"}[1m])) / sum(rate(flask_http_request_duration_seconds_count{job=\"flask-app\"}[1m]))","legendFormat":"{{job}}","range":true}],"fieldConfig":{"defaults":{"unit":"s","min":0},"overrides":[]},"options":{"legend":{"displayMode":"list","placement":"bottom"}}},
    {"id":5,"title":"Requests by HTTP status","type":"timeseries","description":"Cumulative process counters reset on app restart.","datasource":{"type":"prometheus","uid":"prometheus"},"gridPos":{"x":0,"y":16,"w":12,"h":8},"targets":[{"refId":"A","expr":"sum by (status) (flask_http_request_total{job=\"flask-app\"})","legendFormat":"HTTP {{status}}","range":true}],"fieldConfig":{"defaults":{"unit":"short","min":0},"overrides":[]},"options":{"legend":{"displayMode":"list","placement":"bottom"}}}
  ]
}
Enter fullscreen mode Exit fullscreen mode

Top comments (0)