Sanket Patharkar

Posted on Sep 25

Dockerized Monitoring Stack

#devops #docker #infrastructure #monitoring

By Sanket Satish Patharkar – Senior Cloud Operation Engineer
Cloud Infrastructure & DevOps Specialist | 3× AWS Certified

If you've ever SSH'd into a box at 2 AM wondering "is it the database, the cache, or the load balancer?", you already know why a monitoring stack matters.

In this post I'll walk through how we dockerized our whole observability stack — Prometheus, Grafana, and a set of exporters for MongoDB, Redis, NGINX and TCP port checks — into one docker-compose.yml that can be deployed to multiple environments (UAT and Prod) with a single command.

You'll get:

🧱 A block diagram of the architecture
🐳 The Docker Compose setup, explained piece by piece
⚙️ Config for Prometheus, exporters and Grafana provisioning
💻 Every command you need to run, terminal-style
🚀 A Jenkins pipeline for deploy + automatic rollback
🩹 Lessons learned the hard way

All hostnames, IPs, and credentials below are placeholders. Replace them with your own.

1. The Architecture

Here's the big picture. Everything inside the dashed box runs as containers on one monitoring host (an EC2 instance in our case). The exporters reach out to the real infrastructure, Prometheus scrapes the exporters, and Grafana sits on top.

And this is how a change gets from Git to the server:

1.1 How the data actually flows (internals)

Understanding the internals makes every later config decision obvious.

Pull model. Prometheus pulls. Every scrape_interval it sends an HTTP GET /metrics to each target, parses the text exposition format, and appends samples to its TSDB. Exporters are stateless translators: on each scrape they query the backend (MongoDB serverStatus, Redis INFO, NGINX stub_status, a TCP handshake) and render the result as metrics. The only push path is Pushgateway, for short-lived jobs that finish before a scrape could happen.

A scrape, step by step:

relabel_configs run before the scrape, on target metadata (__address__, __meta_ec2_*). This is where the Blackbox trick happens.
metric_relabel_configs run after the scrape, on every sample. Use them to drop high-cardinality series you don't need.
Head block + WAL: the last ~2 hours live in memory, protected by a write-ahead log on the prometheus_data volume. That's why a named volume matters: without it, a container restart loses the WAL and the head block.
up metric: for every target Prometheus synthesises up{job,instance} = 1/0, plus scrape_duration_seconds and scrape_samples_scraped. These are your first debugging tools.

Two kinds of "down":

Metric	Meaning
`up{job="redis-exporter"} == 0`	Prometheus can't reach the exporter
`redis_up == 0`	The exporter is fine, but it can't reach Redis
`probe_success == 0`	Blackbox couldn't open a TCP connection to the target port

Alert on all three separately. They point to different fixes.

2. What's in the Stack

Service	Image	Port	What it does
`prometheus`	`prom/prometheus:v3.14.0`	9090	Scrapes & stores metrics (30 days)
`grafana`	`grafana/grafana:13.2.0`	3000	Dashboards + alerting
`mongodb-exporter`	`percona/mongodb_exporter:0.53.0`	9216	MongoDB server/replica-set metrics
`mongodb-query-exporter`	`ghcr.io/raffis/mongodb-query-exporter:v5.1.0`	9412	Business metrics from Mongo aggregations
`redis-exporter`	`oliver006/redis_exporter:v1.90.0`	9121	Redis memory, clients, keys, latency
`nginx-api-exporter`	`nginx/nginx-prometheus-exporter:1.5.3`	9113	NGINX `stub_status` connections/requests
`blackbox-exporter`	`prom/blackbox-exporter:v0.28.0`	9115	TCP "is the port up?" checks
`pushgateway`	`prom/pushgateway:v1.11.3`	9091	Receives metrics from short-lived jobs
`generate-hash`	`python:3-alpine`	–	Helper: bcrypt hash for Prometheus auth

💡 Pin your image versions. latest will eventually break your dashboards on a random Tuesday.

3. Project Layout

monitoring-stack/
├── docker-compose.yml
├── blackbox/
│   └── blackbox.yml
├── env/
│   ├── uat/            ├── .env  └── env.properties
│   └── prod/        ├── .env  └── env.properties
├── prometheus/
│   ├── web.yml                     # basic-auth users (bcrypt)
│   ├── uat/prometheus-tmpl.yml     # templates with ${VARS}
│   └── prod/prometheus-tmpl.yml
├── query-exporter/
│   └── <env>/config.yaml           # Mongo aggregation → metric
├── grafana/
│   ├── conf/grafana.ini
│   ├── provisioning/dashboards/<env>/dashboards.yml
│   └── dashboards/<env>/**.json    # folder structure = Grafana folders
├── scripts/
│   └── prometheus-healthcheck.sh
├── ssl-certs/
│   └── ca.pem                      # CA for Mongo TLS
└── Jenkinsfile (deploy.groovy)

The key idea: one compose file, many environments. Environment differences live in env/<env>/.env, prometheus/<env>/, query-exporter/<env>/ and grafana/dashboards/<env>/ — never in the compose file itself.

4. Prerequisites on the Monitoring Host

You need Docker Engine and the Compose v2 plugin (docker compose, not the old docker-compose).

Amazon Linux 2023:

ec2-user@monitoring-host:~$ sudo dnf install -y docker git gettext
ec2-user@monitoring-host:~$ sudo systemctl enable --now docker
ec2-user@monitoring-host:~$ sudo usermod -aG docker $USER && newgrp docker

# Compose v2 plugin
ec2-user@monitoring-host:~$ sudo mkdir -p /usr/local/lib/docker/cli-plugins
ec2-user@monitoring-host:~$ sudo curl -SL \
    https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64 \
    -o /usr/local/lib/docker/cli-plugins/docker-compose
ec2-user@monitoring-host:~$ sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose

Ubuntu:

ubuntu@monitoring-host:~$ curl -fsSL https://get.docker.com | sudo sh
ubuntu@monitoring-host:~$ sudo apt-get install -y gettext-base git
ubuntu@monitoring-host:~$ sudo usermod -aG docker $USER && newgrp docker

Verify:

ec2-user@monitoring-host:~$ docker --version
Docker version 27.3.1, build ce12230
ec2-user@monitoring-host:~$ docker compose version
Docker Compose version v2.29.7
ec2-user@monitoring-host:~$ envsubst --version | head -1
envsubst (GNU gettext-runtime) 0.21

gettext gives us envsubst, which we use to render the Prometheus config from a template.

Network / security group checklist:

Direction	Port	Why
Inbound	3000	Grafana UI (ideally behind a reverse proxy / ALB)
Inbound	9090	Prometheus UI (restrict to VPN/office)
Outbound	27017 / 6379 / 80 / 8082 …	Exporters → targets
Outbound	443	Pull images, Slack/email webhooks

If Prometheus uses EC2 service discovery (below), attach an IAM instance role with ec2:DescribeInstances.

5. Dockerizing the Stack — the Compose File, Explained

Let's go through the interesting parts of docker-compose.yml.

5.1 Profiles = environments

Each service declares which environments it belongs to:

services:
  blackbox-exporter:
    image: prom/blackbox-exporter:v0.28.0
    container_name: blackbox-exporter
    restart: always
    profiles: [uat, prod]
    volumes:
      - ./blackbox/blackbox.yml:/etc/blackbox_exporter/config.yml:ro
    command:
      - --config.file=/etc/blackbox_exporter/config.yml
    ports:
      - "9115:9115"

A service with no matching profile is simply not started. That lets UAT run extra QA exporters while prod runs a leaner set — from the same file.

5.2 Same container name, different config per environment

Our production MongoDB is big. Percona's --collect-all includes collstats, which on a large DB starved the connection pool so the important diagnosticdata collector never ran. The fix: two service definitions, same container_name, mutually exclusive profiles.

  # UAT: smaller DB tolerates collect-all
  mongodb-exporter:
    image: percona/mongodb_exporter:0.53.0
    container_name: mongodb-exporter
    restart: always
    profiles: [uat]
    command:
      - --collect-all
    environment:
      - MONGODB_URI=mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD_ENCODED}@${MONGO_DB_HOST}:${MONGO_DB_PORT}/?directConnection=true&authSource=${MONGO_DB_NAME}&tls=true&tlsCAFile=/ssl-certs/ca.pem
    volumes:
      - ./ssl-certs:/ssl-certs:ro
    ports:
      - "9216:9216"
    extra_hosts:
      - "${MONGO_DB_HOST_DNS}"

  # Prod: only the collectors the dashboards need
  mongodb-exporter-prod:
    image: percona/mongodb_exporter:0.53.0
    container_name: mongodb-exporter
    restart: always
    profiles: [prod]
    command:
      - --collector.diagnosticdata
      - --collector.replicasetstatus
      - --collector.dbstats
      - --collector.topmetrics
    # ...same environment / volumes / ports as above

Two things worth copying:

extra_hosts injects hostname:ip into the container's /etc/hosts. Mongo TLS certs are issued for a hostname, so we connect by name even when there's no private DNS. The value comes from .env, e.g. MONGO_DB_HOST_DNS=mongo-primary.internal:x.x.x.x.
MONGO_DB_PASSWORD_ENCODED — URL-encode the password before putting it in a URI. A single @ or / in a password will ruin your afternoon.

5.3 Business metrics straight from MongoDB

Infrastructure metrics tell you the DB is healthy. They don't tell you "success rate for API calls dropped from 99% to 80%". For that we use mongodb-query-exporter, which turns an aggregation pipeline into a Prometheus gauge:

  mongodb-query-exporter:
    image: ghcr.io/raffis/mongodb-query-exporter:v5.1.0
    container_name: mongodb-query-exporter
    restart: always
    profiles: [uat, prod]
    environment:
      - MDBEXPORTER_MONGODB_URI=mongodb://${MONGO_DB_USERNAME}:${MONGO_DB_PASSWORD}@${MONGO_DB_HOST}:${MONGO_DB_PORT}/?authSource=${MONGO_DB_NAME}&tls=true&tlsCAFile=/ssl-certs/ca.pem&readPreference=secondaryPreferred
      - MDBEXPORTER_CONFIG=/config/config.yaml
    volumes:
      - ./query-exporter/${env}/config.yaml:/config/config.yaml:ro
      - ./ssl-certs:/ssl-certs:ro
    ports:
      - "9412:9412"

Note readPreference=secondaryPreferred — never run reporting aggregations on your primary.

query-exporter/<env>/config.yaml (simplified):

version: 3.0
bind: 0.0.0.0:9412
metricsPath: /metrics

global:
  queryTimeout: "120s"
  maxConnection: 1

servers:
  - name: main
    tls: true

aggregations:
  - server: main
    database: appdb
    collection: transactions
    cache: "5m"          # run the pipeline at most every 5 minutes
    mode: pull
    metrics:
      - name: app_total_calls
        type: gauge
        help: "Total calls per tenant (last hour)"
        value: total
        labels: [tenantId]
      - name: app_success_calls
        type: gauge
        help: "Successful calls per tenant (last hour)"
        value: success
        labels: [tenantId]
    pipeline: |
      [
        { "$match": { "createdAt": { "$gte": { "$date": { "$numberLong": "-3600000" } } } } },
        { "$group": {
            "_id": "$tenantId",
            "total":   { "$sum": 1 },
            "success": { "$sum": { "$cond": [{ "$eq": ["$status", "SUCCESS"] }, 1, 0] } }
        } },
        { "$project": { "_id": 0, "tenantId": "$_id", "total": 1, "success": 1 } }
      ]

Now app_success_calls / app_total_calls is a success-rate panel and an alert rule.

5.4 Redis & NGINX exporters

  redis-exporter:
    image: oliver006/redis_exporter:v1.90.0
    container_name: redis-exporter
    restart: always
    profiles: [uat, prod]
    command:
      - --redis.addr=redis://redis.internal:${REDIS_PORT}
      - --redis.password=${REDIS_DB_PASSWORD}
    ports:
      - "9121:9121"
    extra_hosts:
      - "${REDIS_HOST_DNS}"

  nginx-api-exporter:
    image: nginx/nginx-prometheus-exporter:1.5.3
    container_name: nginx-api-exporter
    restart: always
    profiles: [uat, prod]
    command:
      - --nginx.scrape-uri=http://nginx1:8082/stub_status
      - --nginx.scrape-uri=http://nginx2:8082/stub_status
    ports:
      - "9113:9113"
    extra_hosts:
      - "${NGINX_HOST1}"   # nginx1:x.x.x.x
      - "${NGINX_HOST2}"   # nginx2:x.x.x.x

On the NGINX side you need a stub_status location — keep it on an internal-only port:

server {
    listen 8082;
    allow 10.0.0.0/8;
    deny all;
    location /stub_status { stub_status; }
}

5.5 Prometheus — host networking, auth, healthcheck

  prometheus:
    image: prom/prometheus:v3.14.0
    container_name: prometheus
    restart: always
    profiles: [uat, prod]
    network_mode: host
    environment:
      - PROMETHEUS_PASSWORD=${PROMETHEUS_PASSWORD}
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/web.yml:/etc/prometheus/web.yml:ro
      - ./scripts/prometheus-healthcheck.sh:/usr/local/bin/prometheus-healthcheck:ro
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--web.config.file=/etc/prometheus/web.yml'
      - '--storage.tsdb.retention.time=30d'
      - '--web.enable-lifecycle'
      - '--web.listen-address=0.0.0.0:9090'
      - '--web.external-url=${PROMETHEUS_WEB_EXTERNAL_URL}'
    healthcheck:
      test: ["CMD", "/bin/sh", "/usr/local/bin/prometheus-healthcheck"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s
    depends_on:
      - mongodb-query-exporter
      - blackbox-exporter
      - redis-exporter
      - nginx-api-exporter

Why these choices?

network_mode: host — every exporter publishes its port on the host, so Prometheus can scrape them all as localhost:<port>. Simple, and EC2 service discovery hits real private IPs without Docker NAT in the way.
--web.enable-lifecycle — lets you hot-reload config with POST /-/reload instead of restarting.
Named volume prometheus_data — metrics survive docker compose down.
web.yml — Prometheus' built-in basic auth, so the UI and API aren't wide open.

5.6 Grafana — plugins, provisioning, healthcheck

  grafana:
    image: grafana/grafana:13.2.0
    container_name: grafana
    restart: always
    profiles: [uat, prod]
    network_mode: host
    environment:
      - GF_SECURITY_ADMIN_USER=${GF_SECURITY_ADMIN_USER}
      - GF_SECURITY_ADMIN_PASSWORD=${GF_SECURITY_ADMIN_PASSWORD}
      - GF_SERVER_ROOT_URL=${GF_SERVER_ROOT_URL}
      - GF_SERVER_HTTP_PORT=3000
      - GF_SMTP_ENABLED=${GF_SMTP_ENABLED}
      - GF_SMTP_HOST=${GF_SMTP_HOST}
      - GF_SMTP_USER=${GF_SMTP_USER}
      - GF_SMTP_PASSWORD=${GF_SMTP_PASSWORD}
      - GF_INSTALL_PLUGINS=redis-datasource
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/conf/grafana.ini:/etc/grafana/grafana.ini:ro
      - ./grafana/provisioning/dashboards/${env}:/etc/grafana/provisioning/dashboards:ro
      - ./grafana/dashboards/${env}:/etc/grafana/dashboards:ro
    healthcheck:
      test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1"]
      interval: 30s
      timeout: 5s
      retries: 2
    depends_on:
      - prometheus

volumes:
  grafana_data:
  prometheus_data:

Dashboards are file-provisioned — grafana/provisioning/dashboards/<env>/dashboards.yml:

apiVersion: 1
providers:
  - name: 'Default'
    orgId: 1
    type: file
    disableDeletion: false
    updateIntervalSeconds: 10
    allowUiUpdates: true
    options:
      path: /etc/grafana/dashboards
      foldersFromFilesStructure: true   # sub-folders become Grafana folders

Drop a JSON file into grafana/dashboards/<env>/redis/cluster-health.json and it shows up in a redis folder within 10 seconds. Dashboards-as-code, no clicking.

5.7 A "tools" profile for one-off helpers

Need a bcrypt hash for prometheus/web.yml? Don't install Python on the server — run it in a throwaway container:

  generate-hash:
    image: python:3-alpine
    command: >
      sh -c "pip install --quiet bcrypt &&
      python3 -c 'import bcrypt, sys;
      print(bcrypt.hashpw(sys.argv[1].encode(), bcrypt.gensalt(rounds=10)).decode())'
      ${PROMETHEUS_PASSWORD}"
    profiles: [tools]

Because its only profile is tools, it never starts with the normal stack.

6. Configuration Files

6.1 The `.env` file

Non-secret, per-environment values are committed in env/<env>/.env. Secrets are not — they're injected at deploy time (see section 8).

# env/uat/.env  (example – placeholders only)
env=uat
COMPOSE_PROFILES=uat

MONGO_DB_HOST=mongo-primary.internal
MONGO_DB_PORT=27017
MONGO_DB_NAME=admin
MONGO_DB_HOST_DNS=mongo-primary.internal:x.x.x.x

REDIS_HOST_DNS=redis.internal:x.x.x.x
REDIS_PORT=6379

NGINX_HOST1=nginx1:x.x.x.x
NGINX_HOST2=nginx2:x.x.x.x

GF_SERVER_ROOT_URL=https://grafana.example.com
PROMETHEUS_WEB_EXTERNAL_URL=https://prometheus.example.com
PROMETHEUS_AWS_REGION=us-west-2

# Injected at deploy time – NOT committed:
# MONGO_DB_USERNAME, MONGO_DB_PASSWORD, MONGO_DB_PASSWORD_ENCODED,
# REDIS_DB_PASSWORD, PROMETHEUS_PASSWORD, GF_SECURITY_ADMIN_PASSWORD, GF_SMTP_PASSWORD

💡 Setting COMPOSE_PROFILES in .env means plain docker compose up -d automatically picks the right profile.

6.2 Prometheus template

prometheus/<env>/prometheus-tmpl.yml contains ${VARS} that get rendered with envsubst:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    basic_auth:
      username: admin
      password: "${PROMETHEUS_PASSWORD}"
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'grafana'
    static_configs:
      - targets: ['localhost:3000']

  - job_name: 'mongodb-exporter'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:9216']

  - job_name: 'mongodb-query-exporter'
    scrape_interval: 60s
    scrape_timeout: 55s        # aggregations can be slow
    static_configs:
      - targets: ['localhost:9412']

  - job_name: 'redis-exporter'
    scrape_interval: 30s
    static_configs:
      - targets: ['localhost:9121']

  - job_name: 'nginx-api-exporter'
    static_configs:
      - targets: ['localhost:9113']

  # --- Blackbox TCP port checks via EC2 service discovery ---
  - job_name: 'app-port-check'
    metrics_path: /probe
    params:
      module: [tcp_connect]
    ec2_sd_configs:
      - region: ${PROMETHEUS_AWS_REGION}
        port: 8080
        filters:
          - name: tag:cluster
            values: [APP-CLUSTER]
    relabel_configs:
      - source_labels: [__meta_ec2_private_ip]
        replacement: '${1}:8080'
        target_label: __param_target
      - source_labels: [__meta_ec2_tag_Name]
        target_label: instance
      - target_label: __address__
        replacement: localhost:9115   # send the probe to blackbox

The blackbox relabel dance, in plain English:

EC2 SD finds every instance tagged cluster=APP-CLUSTER.
Its private IP becomes the target query param.
The actual scrape goes to localhost:9115/probe?module=tcp_connect&target=x.x.x.x:8080.
You get probe_success{instance="app-1"} = 1 (up) or 0 (down).

New instances in the cluster are monitored automatically — no config change.

6.3 Blackbox module

# blackbox/blackbox.yml
modules:
  tcp_connect:
    prober: tcp
    timeout: 5s

6.4 Prometheus basic auth

# prometheus/web.yml
basic_auth_users:
  admin: "$2b$10$REPLACE_WITH_BCRYPT_HASH"

rounds=10 keeps bcrypt verification around ~50–80 ms. Prometheus checks the hash on every authenticated request (including Grafana's queries), so don't raise it to 14+ unless you enjoy slow dashboards.

6.5 Least-privilege credentials for exporters

Exporters should never connect as an admin. Create dedicated read-only identities.

MongoDB — the clusterMonitor role covers serverStatus, replSetGetStatus and dbStats. The query exporter additionally needs read on the databases it aggregates:

// mongosh, connected to the primary as an admin
use admin
db.createUser({
  user: "prom_exporter",
  pwd:  passwordPrompt(),
  roles: [
    { role: "clusterMonitor", db: "admin" },
    { role: "read",           db: "local" },   // oplog window metrics
    { role: "read",           db: "appdb" }    // query-exporter aggregations
  ]
})

Redis 6+ ACL — allow only the commands redis_exporter uses:

redis-cli ACL SETUSER prom_exporter on '>STRONG_PASSWORD' '~*' '&*' \
  -@all +@connection +memory +info +latency +slowlog +config|get +cluster|info \
  +client +scan +type +strlen +xinfo +xlen +pfcount +zcard +scard +llen +hlen +get +eval \
  -hello -echo -quit -auth -reset -readonly -readwrite -asking -wait

AWS IAM — the instance role only needs to describe instances for EC2 service discovery and read its own secret:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["ec2:DescribeInstances", "ec2:DescribeAvailabilityZones"], "Resource": "*" },
    { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:*:<ACCOUNT_ID>:secret:monitoring/*" }
  ]
}

7. Running the Stack — Step by Step

Here's the full manual flow, exactly as you'd type it.

Step 1 — Clone the repo

ec2-user@monitoring-host:~$ sudo mkdir -p /opt && cd /opt
ec2-user@monitoring-host:/opt$ git clone https://github.com/patharkar123/monitoring-stack.git monitoring
Cloning into 'monitoring'...
remote: Enumerating objects: 1432, done.
Receiving objects: 100% (1432/1432), 2.10 MiB | 8.40 MiB/s, done.
ec2-user@monitoring-host:/opt$ cd monitoring

Step 2 — Build the runtime `.env`

Start from the committed env file, then append secrets (from your secret manager, vault, or — for a lab — by hand):

ec2-user@monitoring-host:/opt/monitoring$ export ENV_NAME=uat
ec2-user@monitoring-host:/opt/monitoring$ cp env/$ENV_NAME/.env .env

ec2-user@monitoring-host:/opt/monitoring$ aws secretsmanager get-secret-value \
    --secret-id monitoring/$ENV_NAME --region us-west-2 \
    --query SecretString --output text \
  | jq -r 'to_entries[] | "\(.key)=\"\(.value)\""' >> .env

ec2-user@monitoring-host:/opt/monitoring$ chmod 600 .env
ec2-user@monitoring-host:/opt/monitoring$ grep -c = .env
58

Step 3 — Generate the Prometheus password hash

ec2-user@monitoring-host:/opt/monitoring$ docker compose --profile tools run --rm generate-hash
[+] Pulling 1/1
 ✔ generate-hash Pulled                                                     3.2s
$2b$10$Qm8x0l3d0Vb7gJ1tq5o1KeY8W2r3oNQ3m2xF1c9bZ0t4pF7yXh6uS

Paste that hash into prometheus/web.yml.

Step 4 — Render `prometheus.yml` from the template

ec2-user@monitoring-host:/opt/monitoring$ (set -a; source .env; set +a; \
    envsubst < prometheus/$ENV_NAME/prometheus-tmpl.yml > prometheus/prometheus.yml)

ec2-user@monitoring-host:/opt/monitoring$ docker run --rm --entrypoint promtool \
    -v $PWD/prometheus:/p prom/prometheus:v3.14.0 check config /p/prometheus.yml
Checking /p/prometheus.yml
 SUCCESS: /p/prometheus.yml is valid prometheus config file syntax

✅ Always run promtool check config before starting. A YAML typo otherwise shows up as a crash-looping container.

Step 5 — Validate the compose file

ec2-user@monitoring-host:/opt/monitoring$ docker compose --profile $ENV_NAME config --services
mongodb-exporter
mongodb-query-exporter
blackbox-exporter
redis-exporter
nginx-api-exporter
pushgateway
prometheus
grafana

This shows exactly which services the profile will start — a great sanity check.

Step 6 — Start everything

ec2-user@monitoring-host:/opt/monitoring$ docker compose -p monitoring --profile $ENV_NAME up -d
[+] Running 11/11
 ✔ Network monitoring_default            Created                            0.1s
 ✔ Volume "monitoring_prometheus_data"   Created                            0.0s
 ✔ Volume "monitoring_grafana_data"      Created                            0.0s
 ✔ Container blackbox-exporter           Started                            0.8s
 ✔ Container redis-exporter              Started                            0.8s
 ✔ Container mongodb-exporter            Started                            0.9s
 ✔ Container mongodb-query-exporter      Started                            0.9s
 ✔ Container nginx-api-exporter          Started                            0.8s
 ✔ Container pushgateway                 Started                            0.7s
 ✔ Container prometheus                  Started                            1.2s
 ✔ Container grafana                     Started                            1.6s

Step 7 — Check container health

ec2-user@monitoring-host:/opt/monitoring$ docker compose -p monitoring ps --format "table {{.Name}}\t{{.Status}}\t{{.Ports}}"
NAME                     STATUS                    PORTS
blackbox-exporter        Up 2 minutes              0.0.0.0:9115->9115/tcp
grafana                  Up 2 minutes (healthy)
mongodb-exporter         Up 2 minutes              0.0.0.0:9216->9216/tcp
mongodb-query-exporter   Up 2 minutes              0.0.0.0:9412->9412/tcp
nginx-api-exporter       Up 2 minutes              0.0.0.0:9113->9113/tcp
prometheus               Up 2 minutes (healthy)
pushgateway              Up 2 minutes              0.0.0.0:9091->9091/tcp
redis-exporter           Up 2 minutes              0.0.0.0:9121->9121/tcp

(Prometheus and Grafana show no ports because they use host networking.)

Step 8 — Verify each exporter

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9216/metrics | grep -m1 '^mongodb_up'
mongodb_up{cluster_role="mongod"} 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9121/metrics | grep '^redis_up'
redis_up 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9113/metrics | grep '^nginx_up'
nginx_up 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s "localhost:9115/probe?module=tcp_connect&target=x.x.x.x:27017" | grep '^probe_success'
probe_success 1

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:9412/metrics | grep -m2 '^app_'
app_total_calls{tenantId="tenant-a"} 1824
app_success_calls{tenantId="tenant-a"} 1797

Step 9 — Verify Prometheus targets

ec2-user@monitoring-host:/opt/monitoring$ curl -s -u admin:$PROMETHEUS_PASSWORD \
    localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | "\(.health)\t\(.labels.job)"' | sort | uniq -c
      1 up  grafana
      4 up  app-port-check
      1 up  mongodb-exporter
      1 up  mongodb-query-exporter
      1 up  nginx-api-exporter
      1 up  prometheus
      1 up  redis-exporter

Anything showing down? Check .lastError on that target:

ec2-user@monitoring-host:/opt/monitoring$ curl -s -u admin:$PROMETHEUS_PASSWORD localhost:9090/api/v1/targets \
  | jq -r '.data.activeTargets[] | select(.health!="up") | "\(.labels.job): \(.lastError)"'

Step 10 — Open Grafana

ec2-user@monitoring-host:/opt/monitoring$ curl -s localhost:3000/api/health
{
  "database": "ok",
  "version": "13.2.0"
}

Browse to http://<host>:3000, log in with GF_SECURITY_ADMIN_USER / GF_SECURITY_ADMIN_PASSWORD, add Prometheus (http://localhost:9090, basic auth) as a data source, and your provisioned dashboards are already waiting in their folders.

8. PromQL, Recording Rules & Alerting

Collecting metrics is half the job. These are the queries that actually power the dashboards and alerts.

8.1 Queries worth knowing

# Business success rate per tenant (%), from mongodb-query-exporter
100 * app_success_calls / clamp_min(app_total_calls, 1)

# MongoDB replication lag per secondary (seconds; optimeDate is in ms)
(scalar(max(mongodb_rs_members_optimeDate{member_state="PRIMARY"}))
  - mongodb_rs_members_optimeDate{member_state="SECONDARY"}) / 1000

# MongoDB connection saturation (%)
100 * mongodb_ss_connections{conn_type="current"}
    / (mongodb_ss_connections{conn_type="current"} + mongodb_ss_connections{conn_type="available"})

# Redis memory usage vs maxmemory (%)
100 * redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1)

# Redis cache hit ratio (5m)
rate(redis_keyspace_hits_total[5m])
  / (rate(redis_keyspace_hits_total[5m]) + rate(redis_keyspace_misses_total[5m]))

# NGINX requests per second across the cluster
sum(rate(nginx_http_requests_total[1m]))

# Which ports are down right now?
probe_success == 0

# Exporter itself unreachable
up == 0

💡 clamp_min(x, 1) avoids divide-by-zero NaN when a tenant or instance has no traffic. NaN never triggers alerts, which silently hides real outages.

8.2 Recording rules

Heavy queries evaluated on every dashboard refresh waste CPU. Pre-compute them with recording rules, then point dashboards at the recorded series.

# prometheus/rules/recording.yml
groups:
  - name: app-sli
    interval: 1m
    rules:
      - record: tenant:success_ratio:pct
        expr: 100 * app_success_calls / clamp_min(app_total_calls, 1)

      - record: cluster:nginx_requests:rate1m
        expr: sum(rate(nginx_http_requests_total[1m]))

      - record: instance:redis_memory_used:pct
        expr: 100 * redis_memory_used_bytes / clamp_min(redis_memory_max_bytes, 1)

Load them by adding rule_files: ['/etc/prometheus/rules/*.yml'] to the template and mounting ./prometheus/rules:/etc/prometheus/rules:ro. The naming convention level:metric:operation tells readers at a glance what was aggregated.

8.3 Alert rules

We manage alerts in Grafana (provisioned from JSON via its API), but the logic is identical in native Prometheus rule syntax:

groups:
  - name: availability
    rules:
      - alert: ServicePortDown
        expr: probe_success == 0
        for: 2m                     # ignore single failed probes
        labels: { severity: critical }
        annotations:
          summary: "Port check failing on {{ $labels.instance }}"
          description: "TCP connect to {{ $labels.probe_target }} failed for 2 minutes."

      - alert: MongoReplicationLagHigh
        expr: |
          (scalar(max(mongodb_rs_members_optimeDate{member_state="PRIMARY"}))
            - mongodb_rs_members_optimeDate{member_state="SECONDARY"}) / 1000 > 30
        for: 5m
        labels: { severity: warning }

      - alert: TenantSuccessRateDrop
        expr: tenant:success_ratio:pct < 90 and app_total_calls > 50
        for: 10m
        labels: { severity: critical }
        annotations:
          summary: "Success rate for {{ $labels.tenantId }} is {{ $value | printf \"%.1f\" }}%"

Design notes:

for: is your noise filter. Port checks: 1–2 minutes. Capacity alerts: 10+ minutes.
and app_total_calls > 50 prevents alerting on a 1-of-2 failure at 3 AM when traffic is near zero.
Route severity=critical to on-call chat/pager and warning to email. Everyone ignores a channel that pages for everything.

9. Capacity Planning & Hardening

9.1 Sizing Prometheus storage

Prometheus stores roughly 1–2 bytes per sample after compression. Estimate disk with:

disk_bytes ≈ retention_seconds × ingested_samples_per_second × bytes_per_sample

Worked example for this stack:

Input	Value
Active series	~60,000
Average scrape interval	30 s
Samples/second	60,000 / 30 = 2,000
Retention	30 d = 2,592,000 s
Bytes/sample	~1.5
Disk	2,592,000 × 2,000 × 1.5 ≈ 7.8 GB (+ WAL & compaction headroom → provision 20 GB)

Check your real numbers instead of guessing:

ec2-user@monitoring-host:~$ curl -s -u admin:$PROMETHEUS_PASSWORD localhost:9090/api/v1/status/tsdb \
    | jq '.data.headStats, (.data.seriesCountByMetricName[:5])'
{
  "numSeries": 58312,
  "chunkCount": 121480,
  "minTime": 1790370000000,
  "maxTime": 1790377200000
}
[
  { "name": "mongodb_ss_wt_cache_bytes", "value": 4210 },
  { "name": "mongodb_collstats_storageStats_indexSizes", "value": 3877 },
  ...
]

ec2-user@monitoring-host:~$ curl -s -u admin:$PROMETHEUS_PASSWORD \
    'localhost:9090/api/v1/query?query=rate(prometheus_tsdb_head_samples_appended_total[5m])' \
    | jq -r '.data.result[0].value[1]'
1987.4

The seriesCountByMetricName list is your cardinality hit list. If one metric dominates, drop it with metric_relabel_configs:

  - job_name: 'mongodb-exporter'
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'mongodb_collstats_storageStats_indexSizes'
        action: drop

Memory rule of thumb: ~3–4 KB of RAM per active series in the head block. 60k series ≈ 250 MB, so a t3.medium (4 GB) runs this whole stack comfortably.

9.2 Container resource limits & log rotation

Without limits, one misbehaving exporter can starve Prometheus. Without log rotation, Docker's json-file logs will eventually fill the disk.

x-defaults: &defaults
  restart: always
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "3" }

services:
  redis-exporter:
    <<: *defaults
    image: oliver006/redis_exporter:v1.90.0
    mem_limit: 128m
    cpus: 0.25

  prometheus:
    <<: *defaults
    image: prom/prometheus:v3.14.0
    mem_limit: 2g
    cpus: 1.0

The x- extension field + YAML anchor (&defaults / <<: *defaults) keeps these settings in one place.

9.3 Network hardening

Prometheus and Grafana use host networking, so they reach exporters via localhost. That means exporter ports don't need to be exposed on the network interface at all. Bind them to loopback:

    ports:
      - "127.0.0.1:9216:9216"   # instead of "9216:9216"

Verify what's actually listening externally:

ec2-user@monitoring-host:~$ sudo ss -tlnp | awk 'NR==1 || /:(3000|9090|91[0-9][0-9]|92[0-9][0-9]|94[0-9][0-9])/'
State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port  Process
LISTEN  0       4096        127.0.0.1:9216        0.0.0.0:*      users:(("docker-proxy",pid=2211))
LISTEN  0       4096        127.0.0.1:9121        0.0.0.0:*      users:(("docker-proxy",pid=2240))
LISTEN  0       4096                *:9090              *:*      users:(("prometheus",pid=2302))
LISTEN  0       4096                *:3000              *:*      users:(("grafana",pid=2355))

Then put Grafana and Prometheus behind a TLS-terminating load balancer or reverse proxy, and restrict the security group so ports 3000/9090 accept traffic only from that proxy.

⚠️ Docker publishes ports by writing its own iptables rules, which bypass ufw/firewalld. Loopback binding or security groups are the reliable controls, not the host firewall.

10. Automating It — Jenkins Deploy with Rollback

Doing the above by hand is fine once. For every change we use a Jenkins pipeline with one parameter: which environment.

pipeline {
  agent { label "${whereTo}-jenkins-agent" }
  parameters {
    choice(name: 'whereTo', choices: ['uat', 'prod'], description: 'Which env?')
    string(name: 'branch',  defaultValue: 'main')
    booleanParam(name: 'skipDeploy', defaultValue: false)
  }
  stages {
    stage('checkout') {
      steps { dir('monitoring') { git url: env.REPO_URL, branch: params.branch, credentialsId: 'git-ssh' } }
    }
    stage('init') {
      steps { dir('monitoring') { script {
        def envVars = readFile("env/${whereTo}/.env")
        def props   = readProperties file: "env/${whereTo}/env.properties"   // hostIps=x.x.x.x,...
        hostIps     = props.hostIps.split(',')

        // secrets → appended to .env (never committed)
        def secret = readJSON text: getSecret("monitoring/${whereTo}")
        secret.each { k, v -> envVars += "\n${k}=\"${v}\"" }
        writeFile file: '.env', text: envVars

        sh """(set +x; set -a; source .env; set +a;
               envsubst < prometheus/${whereTo}/prometheus-tmpl.yml > prometheus/prometheus.yml)"""
      }}}
    }
    stage('deploy') {
      when { expression { !params.skipDeploy } }
      steps { script {
        hostIps.each { ip ->
          try { doRelease(ip) }
          catch (e) { echo "❌ Release failed on ${ip}: ${e} — rolling back"; doRelease(ip, true); error(e.message) }
        }
      }}
    }
  }
}

def doRelease(ip, rollback = false) {
  def live = '/opt/monitoring', bk = '/opt/monitoring-bk'
  if (rollback) {
    ssh(ip, "sudo rm -rf ${live} && sudo mv ${bk} ${live}")
  } else {
    ssh(ip, "sudo bash -c 'mkdir -p ${live} && rm -rf ${bk} /tmp/monitoring && mv ${live} ${bk}'")
    scp(ip, 'monitoring', '/tmp')                 // push the rendered workspace
    ssh(ip, "sudo mv /tmp/monitoring ${live}")
  }
  ssh(ip, "sudo bash -c 'cd ${live} && source .env && docker compose -p monitoring down || true && docker compose up -d'")
}

The pattern is simple but effective:

   live dir ──mv──► backup dir          (keep last good release)
   new files ──scp──► /tmp ──mv──► live dir
   docker compose down && up -d
        │
        ├─ success ─► done ✅
        └─ failure ─► rm live, mv backup → live, compose up ─► previous version running ↩️

Because the rendered .env and prometheus.yml travel with the release, the rollback restores both code and config.

11. Day-2 Operations Cheat Sheet

# Tail logs for one service
$ docker compose -p monitoring logs -f --tail=100 prometheus

# Hot-reload Prometheus after editing prometheus.yml (no restart)
$ curl -X POST -u admin:$PROMETHEUS_PASSWORD localhost:9090/-/reload

# Restart a single exporter
$ docker compose -p monitoring restart mongodb-query-exporter

# Upgrade an image: bump the tag in docker-compose.yml, then
$ docker compose -p monitoring pull grafana
$ docker compose -p monitoring up -d grafana

# See resource usage
$ docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# How big is the TSDB?
$ docker system df -v | grep prometheus_data

# Push a metric from a cron/batch job
$ echo "backup_last_success_timestamp $(date +%s)" \
    | curl --data-binary @- localhost:9091/metrics/job/nightly_backup

# Stop everything (volumes – and your data – are kept)
$ docker compose -p monitoring down

# ⚠️ Stop AND delete all metrics & Grafana data
$ docker compose -p monitoring down -v

# Clean up old images after upgrades
$ docker image prune -f

12. Troubleshooting Guide

Symptom	Likely cause	How to confirm	Fix
`up{job="mongodb-query-exporter"} == 0` intermittently	Aggregation slower than `scrape_timeout`	`scrape_duration_seconds` ≈ timeout	Add indexes, raise `cache`, keep `scrape_timeout` < `scrape_interval`
`mongodb_up 0`, exporter `up 1`	TLS hostname mismatch / wrong `extra_hosts`	`docker logs mongodb-exporter` → `x509: certificate is valid for …`	Connect by the cert's hostname; fix `*_HOST_DNS`
`mongodb_ss_*` series missing	`collstats` starving the connection pool	Only `mongodb_collstats_*` present	Use explicit `--collector.*` flags
Compose: `service "x" depends on undefined service`	`depends_on` a service outside the active profile	`docker compose --profile <env> config`	Only depend on services present in every profile
Prometheus restarts in a loop	Invalid rendered `prometheus.yml` (empty `${VAR}`)	`promtool check config`	Ensure all template variables exist in `.env`
Grafana "401" on Prometheus datasource	Hash in `web.yml` ≠ password in datasource	`curl -u admin:<pw> localhost:9090/-/healthy`	Regenerate hash, restart Prometheus
Disk full on host	Unrotated container logs or TSDB growth	`docker system df -v`, `du -sh /var/lib/docker/containers/*`	Log rotation (9.2), drop high-cardinality metrics
`probe_success 0` but service works from your laptop	Security group blocks the monitoring host	`nc -vz <ip> <port>` from the host	Allow the monitoring host's SG as source

13. Lessons Learned (the Hard Way)

1. depends_on + profiles can break the whole project.
If Prometheus depends_on: mongodb-exporter-prod but you run the uat profile, Compose treats that service as undefined and refuses to load the project. Only depend on services that exist in every profile the dependent runs in.

2. --collect-all is not free.
On large MongoDB clusters, collstats monopolised the exporter's connection and critical metrics (mongodb_ss_*, replica-set state) went missing. Enable only the collectors your dashboards use.

3. Point business queries at secondaries and cache them.
readPreference=secondaryPreferred + cache: "5m" + a generous scrape_timeout kept heavy aggregations away from production traffic.

4. Use extra_hosts for TLS hostnames.
TLS verification needs the hostname on the certificate. Injecting host:ip via extra_hosts avoids hacking /etc/hosts on the server itself.

5. Keep secrets out of Git, render at deploy time.
Committed .env = non-secret config. Secrets are fetched from a secret manager and appended in the pipeline workspace. The Prometheus password is envsubst-ed into prometheus.yml only on the deploy host.

6. Validate before you restart.
docker compose config, promtool check config, and a --dry-run/skipDeploy flag in the pipeline catch 90% of mistakes before they take monitoring down — which is the one system you really need up when everything else is on fire.

7. Know who owns what in Grafana.
If you mix file provisioning, API provisioning and Grafana's Git Sync, make each one own a clearly separate set of resources (e.g. dashboards via files/Git, alerts & datasources via API). Overlap leads to "who deleted my datasource?" mysteries.

Wrapping Up

With one docker-compose.yml, a handful of config files and a small pipeline, we got:

✅ Infra metrics for MongoDB, Redis, NGINX
✅ Business KPIs straight from MongoDB aggregations
✅ Automatic port checks for every instance in a cluster via EC2 discovery
✅ Dashboards-as-code with folder structure
✅ Repeatable, multi-environment deploys with automatic rollback

The whole stack comes up in under a minute on a fresh host:

The full stack is in patharkar123/monitoring-stack:

$ git clone https://github.com/patharkar123/monitoring-stack.git monitoring
$ cd monitoring
$ cp env/uat/.env .env   # + secrets
$ envsubst < prometheus/uat/prometheus-tmpl.yml > prometheus/prometheus.yml
$ docker compose --profile uat up -d

If you found this useful, drop a ❤️ or a comment — and tell me what you monitor that I didn't cover here. For any issue or support, reach me on LinkedIn. Happy monitoring! 📈

🔜 Up Next

In the next article I'll cover alerting as code: provisioning Grafana alert rules, contact points and datasources through the Grafana HTTP API from Jenkins, with idempotent diffing and dry-run support.

👤 About the Author

Sanket Satish Patharkar is a Senior Cloud Operation Engineer with over 6 years of experience in AWS, DevOps, CI/CD, and cloud architecture for enterprise-grade systems. 3× AWS Certified, with a focus on cloud infrastructure, observability, automation, and GenAI.

📍 Pune, Maharashtra, India · 🎓 B.E. in Electronics & Telecommunication, Savitribai Phule Pune University

Repo: github.com/patharkar123/monitoring-stack · For any issue or support: LinkedIn