DEV Community: Airat Iunusov

I tested my own docs like a stranger. They failed 24 times.

Airat Iunusov — Thu, 15 Oct 2026 20:08:52 +0000

I wrote a monitoring stack (Prometheus, Grafana, Alertmanager) and a README that says "it just works". It had always worked on my own server. But I had set that one up by hand, over time, from memory. I had never once followed my own README from the first line to the last.

So I took a clean VM and did exactly that, the way a newcomer would. First on Ubuntu 22.04, then on 24.04 and 26.04. It took about a day.

I found 24 problems. Three of them stuck with me, because I'm the one who got them wrong.

The alert that fired on a healthy machine

On Ubuntu 22.04 a fresh VM with nothing running went to pending on a critical alert: FilesystemReadonly. My first thought was that I'd broken something.

It was my rule. systemd 249 mounts a read-only ramfs for its own needs, and my alert counted it as a failing disk. Ubuntu 24.04 and 26.04 don't do this, so on newer systems I'd never have noticed.

The fix is one word: add ramfs to the exclusion list. The more interesting part is that my own checks had missed it. I was verifying that targets are up and dashboards load. I wasn't verifying that a healthy system is quiet. Now /api/v1/alerts must be empty on every fresh stand.

The restore that restores nothing

My backup doc explained how to make a backup. Restore was one sentence without commands. So I tried it.

The Grafana archive has a top-level grafana/ directory, the Prometheus one doesn't. Unpack Grafana to the wrong path and nothing fails. You keep the old data and believe you restored.

I wrote the steps down and ran them. To be sure it really restored something, I created a folder with a marker in Grafana before the backup. After the restore it was there, and all 7 targets were up.

A backup you haven't restored is a hope, not a backup.

"Ansible 2.9+"

The README said Ansible 2.9 or newer. I had never run it on 2.9, I just wrote the number.

I tried. On 2.9 the syntax check passes, then the run fails on the target with ModuleNotFoundError, a known clash with Python 3.12. From 2.12 to the latest 2.21 everything worked, and the second run gave changed=0. The README now says 2.12+.

Everything else

The other 21 are smaller: a compose block in the wrong place, a stale link, flags in an example config that no code reads, Alertmanager needing restart instead of up -d. All fixed in 15 pull requests, #26 to #40.

What I did not check: a real x86_64 machine (I used emulation), Debian and RHEL, load.

A check I ran by hand once is a snapshot. The next version bump will break something again, so the next step is a CI smoke test: compose up, targets up, Ansible twice with changed=0. Docs should run like code.

How do you check that your install docs still work: CI, a manual run before a release, or you find out from an issue?

Repo: github.com/Airat71/monitoring-stack

Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.

See who's attacking your server in Grafana — fail2ban metrics with Prometheus

Airat Iunusov — Sat, 03 Oct 2026 14:45:12 +0000

Most people set up fail2ban and forget about it. It runs and bans IPs, but you only know what it does if you SSH in and read the logs.

This post adds fail2ban metrics to a Prometheus stack and shows them in Grafana, on the same board as CPU, memory and disk.

What you will see

Current bans and failed attempts per jail
Total bans over time
Which jail fires most: SSH, Nginx or recidive
Whether the exporter can still reach fail2ban at all

The exporter gives counts and jail names. It does not put banned IP addresses into metrics, so you see how much is happening, not who.

Step 1: check the socket

The exporter reads the fail2ban control socket, not the log files.

test -S /var/run/fail2ban/fail2ban.sock && ls -l /var/run/fail2ban/fail2ban.sock

Stop here if that path is missing or is a directory. Expected mode is owner-only, for example srw------- root root.

Step 2: add the exporter

Add this block to the services: section of your docker-compose.yml:

fail2ban-exporter:
  image: registry.gitlab.com/hctrdev/fail2ban-prometheus-exporter:0.10.3
  container_name: fail2ban-exporter
  restart: unless-stopped
  ports:
    - "127.0.0.1:9191:9191"
  volumes:
    - /var/run/fail2ban:/var/run/fail2ban:ro
  read_only: true
  security_opt:
    - no-new-privileges:true
  networks:
    - monitoring-net
  deploy:
    resources:
      limits:
        memory: 32M

Two things I learned the hard way.

Mount the directory, not the socket file. If the file is missing when the container is created, Docker creates a directory at that path and fail2ban can no longer bind its socket.

This image runs as uid 0, so the socket stays owner-only and I limit the container instead: read-only mount, read_only, no-new-privileges, port on localhost.

Step 3: add a scrape job to Prometheus

In prometheus.yml:

- job_name: 'fail2ban'
  static_configs:
    - targets: ['fail2ban-exporter:9191']
      labels:
        instance: 'your-server'
  scrape_interval: 30s
  scrape_timeout: 10s

Prometheus and the exporter must be on the same Compose network. Reload Prometheus, then check:

docker compose up -d fail2ban-exporter
curl -s http://127.0.0.1:9191/metrics | grep '^f2b_up'

f2b_up must be 1. The Prometheus target can show UP while f2b_up is 0: the process is serving metrics but cannot use the socket.

Step 4: add the alerts

- alert: Fail2banHighAttackRate
  expr: increase(f2b_jail_banned_total[5m]) > 10
  labels:
    severity: warning
  annotations:
    summary: "High fail2ban ban rate on {{ $labels.instance }}"

- alert: Fail2banSocketDown
  expr: f2b_up == 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "fail2ban exporter cannot read the socket on {{ $labels.instance }}"

The first one fires when more than 10 IPs are banned in 5 minutes. The second one covers the case above: the exporter is up but blind.

Step 5: import the dashboard

Download fail2ban.json from grafana-dashboards/json/ in the repo and import it in Grafana via Dashboards, Import. Select the fail2ban job in the $job dropdown.

Why bother

The first time I saw a ban spike in Grafana, 40 IPs banned in under 2 minutes at 3am, I realized I had had no view of this before. It was happening, I just could not see it.

One gotcha: by default systemd removes /var/run/fail2ban when fail2ban stops and creates a new directory on start. A container that mounted the old one keeps a deleted directory, and f2b_up stays 0. A systemd drop-in with RuntimeDirectoryPreserve=yes fixes it. I checked it on Ubuntu 24.04 (arm64) with fail2ban 1.0.2.

What I did not check: other distributions and other fail2ban versions.

Do you export fail2ban metrics, or do you still read the logs?

Repo: github.com/Airat71/monitoring-stack (guide: docs/FAIL2BAN_ENHANCED.md)

Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.