<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Airat Iunusov</title>
    <description>The latest articles on DEV Community by Airat Iunusov (@airat71).</description>
    <link>https://dev.to/airat71</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4159937%2F1d9b076b-9692-4e30-aef4-29382e3d5d11.png</url>
      <title>DEV Community: Airat Iunusov</title>
      <link>https://dev.to/airat71</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/airat71"/>
    <language>en</language>
    <item>
      <title>I installed my stack on a machine with leftovers. Three green signals stayed green.</title>
      <dc:creator>Airat Iunusov</dc:creator>
      <pubDate>Sun, 08 Nov 2026 17:47:20 +0000</pubDate>
      <link>https://dev.to/airat71/i-installed-my-stack-on-a-machine-with-leftovers-three-green-signals-stayed-green-5211</link>
      <guid>https://dev.to/airat71/i-installed-my-stack-on-a-machine-with-leftovers-three-green-signals-stayed-green-5211</guid>
      <description>&lt;p&gt;After &lt;a href="https://dev.to/airat71/i-tested-my-own-docs-like-a-stranger-they-failed-24-times-1ekf"&gt;the first article&lt;/a&gt; a reader (&lt;a class="mentioned-user" href="https://dev.to/devenquan"&gt;@devenquan&lt;/a&gt;) wrote that they had run the same exercise on their own tool and found two problems. Their suggestion: do it once more, on a machine that already has stuff on it. Fair point. I had never tested it that way, so I ran it.&lt;/p&gt;

&lt;p&gt;I used an Ubuntu 24.04 VM (arm64) with Docker 29.9.2 and followed my install docs the way a newcomer would. Five scenarios: busy ports, an old Grafana volume with a different password, an upgrade over existing data, Compose and Ansible on one host, and a &lt;code&gt;node_exporter&lt;/code&gt; from apt already holding port 9100.&lt;/p&gt;

&lt;p&gt;Before the run I wrote down what I expected: the docs cover none of these. They covered none. Two of the five I staged myself.&lt;/p&gt;

&lt;p&gt;My CI starts the same stack on a clean runner. There the exit codes and &lt;code&gt;changed=0&lt;/code&gt; are green and the stack is up. On this VM they were green too, and the stack was not. Green output alone does not tell you whether the stack is up.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Healthy" with no network
&lt;/h2&gt;

&lt;p&gt;I took ports 9090 and 3001 and ran &lt;code&gt;docker compose up -d&lt;/code&gt;. It fails on the first port, three containers run, two stay in &lt;code&gt;Created&lt;/code&gt;. Then I ran it again, because that is what a newcomer does. Prometheus showed &lt;code&gt;Up (healthy)&lt;/code&gt; with no network and no published port. On the host, &lt;code&gt;127.0.0.1:9090&lt;/code&gt; answered with the other process.&lt;/p&gt;

&lt;p&gt;My healthcheck requests &lt;code&gt;localhost&lt;/code&gt; from inside the container, so it can't fail. And plain &lt;code&gt;docker compose ps&lt;/code&gt; hides containers in &lt;code&gt;Created&lt;/code&gt;: the Quick Start told you to run exactly that. This was on Docker 29.9.2 and 29.1.3. On 27.5.1 the container stays in &lt;code&gt;Created&lt;/code&gt;, and I did not test 28.x. So not a Docker bug: the check that can't fail is mine.&lt;/p&gt;

&lt;p&gt;The same shape in Ansible. A &lt;code&gt;prometheus-node-exporter&lt;/code&gt; from apt held port 9100. The first playbook run failed on the bind. The second one exited 0 with &lt;code&gt;ok=19 changed=0 failed=0&lt;/code&gt;, and the &lt;code&gt;node-exporter&lt;/code&gt; target was &lt;code&gt;down&lt;/code&gt; (Docker 29.9.2). In part 1 I used &lt;code&gt;changed=0&lt;/code&gt; as a check. It proves the run is idempotent, not that anything is running.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;smoke-check.sh&lt;/code&gt; failed in both states. The docs just never mentioned it.&lt;/p&gt;

&lt;h2&gt;
  
  
  An upgrade that exits 0
&lt;/h2&gt;

&lt;p&gt;I upgraded from v2.2.0 to v2.3.0 over existing volumes, the obvious way: check out the new tag, &lt;code&gt;docker compose pull&lt;/code&gt;, &lt;code&gt;docker compose up -d&lt;/code&gt;. Data survived and all 5 targets were up. But Grafana showed 13 dashboards instead of 9: three new ones were not applied, their old versions stayed, plus 4 stale ones. Every 30 seconds the log said &lt;code&gt;failed to save dashboard&lt;/code&gt;. &lt;code&gt;up -d&lt;/code&gt; exited 0.&lt;/p&gt;

&lt;p&gt;It is exactly the three files whose names stayed while the &lt;code&gt;uid&lt;/code&gt; inside changed. That fits, but I did not confirm it in Grafana's code. The only fix I verified is deleting the Grafana volume, which also deletes users and anything made in the UI. This one shipped in my own release, v2.3.0.&lt;/p&gt;

&lt;p&gt;Two more, in a line each: a new &lt;code&gt;GRAFANA_PASSWORD&lt;/code&gt; is ignored after the first start, and Compose and Ansible on one host collide on container names and leave a network, three volumes and &lt;code&gt;/opt/monitoring&lt;/code&gt; behind.&lt;/p&gt;

&lt;h2&gt;
  
  
  What changed
&lt;/h2&gt;

&lt;p&gt;It is in &lt;a href="https://github.com/Airat71/monitoring-stack/releases/tag/v2.3.1" rel="noopener noreferrer"&gt;v2.3.1&lt;/a&gt; (&lt;a href="https://github.com/Airat71/monitoring-stack/pull/45" rel="noopener noreferrer"&gt;PR #45&lt;/a&gt;): the docs verify with &lt;code&gt;smoke-check.sh&lt;/code&gt; instead of &lt;code&gt;docker compose ps&lt;/code&gt;, &lt;code&gt;TROUBLESHOOTING.md&lt;/code&gt; has these cases, and a new CI job upgrades the latest release over its own data and checks the dashboards. The release is docs and CI only. I did not touch the healthcheck, the playbook still does not check targets after it starts, and the dashboard bug itself is not fixed: the docs now tell you how to spot it and how to get around it.&lt;/p&gt;

&lt;p&gt;Not checked: Docker 28.x, x86_64, Debian and RHEL, the Ansible upgrade path, an alert reaching Telegram. All of the above is one VM.&lt;/p&gt;

&lt;p&gt;Do you test installs on a machine that already has leftovers, or only on a fresh one? What do you leave on it?&lt;/p&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/Airat71/monitoring-stack" rel="noopener noreferrer"&gt;github.com/Airat71/monitoring-stack&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;An AI assistant helped with this article.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>prometheus</category>
      <category>documentation</category>
    </item>
    <item>
      <title>I tested my own docs like a stranger. They failed 24 times.</title>
      <dc:creator>Airat Iunusov</dc:creator>
      <pubDate>Thu, 15 Oct 2026 20:08:52 +0000</pubDate>
      <link>https://dev.to/airat71/i-tested-my-own-docs-like-a-stranger-they-failed-24-times-1ekf</link>
      <guid>https://dev.to/airat71/i-tested-my-own-docs-like-a-stranger-they-failed-24-times-1ekf</guid>
      <description>&lt;p&gt;I wrote a monitoring stack (Prometheus, Grafana, Alertmanager) and a README that says "it just works". It had always worked on my own server. But I had set that one up by hand, over time, from memory. I had never once followed my own README from the first line to the last.&lt;/p&gt;

&lt;p&gt;So I took a clean VM and did exactly that, the way a newcomer would. First on Ubuntu 22.04, then on 24.04 and 26.04. It took about a day.&lt;/p&gt;

&lt;p&gt;I found 24 problems. Three of them stuck with me, because I'm the one who got them wrong.&lt;/p&gt;

&lt;h2&gt;
  
  
  The alert that fired on a healthy machine
&lt;/h2&gt;

&lt;p&gt;On Ubuntu 22.04 a fresh VM with nothing running went to &lt;code&gt;pending&lt;/code&gt; on a critical alert: &lt;code&gt;FilesystemReadonly&lt;/code&gt;. My first thought was that I'd broken something.&lt;/p&gt;

&lt;p&gt;It was my rule. systemd 249 mounts a read-only &lt;code&gt;ramfs&lt;/code&gt; for its own needs, and my alert counted it as a failing disk. Ubuntu 24.04 and 26.04 don't do this, so on newer systems I'd never have noticed.&lt;/p&gt;

&lt;p&gt;The fix is one word: add &lt;code&gt;ramfs&lt;/code&gt; to the exclusion list. The more interesting part is that my own checks had missed it. I was verifying that targets are up and dashboards load. I wasn't verifying that a healthy system is &lt;em&gt;quiet&lt;/em&gt;. Now &lt;code&gt;/api/v1/alerts&lt;/code&gt; must be empty on every fresh stand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The restore that restores nothing
&lt;/h2&gt;

&lt;p&gt;My backup doc explained how to make a backup. Restore was one sentence without commands. So I tried it.&lt;/p&gt;

&lt;p&gt;The Grafana archive has a top-level &lt;code&gt;grafana/&lt;/code&gt; directory, the Prometheus one doesn't. Unpack Grafana to the wrong path and nothing fails. You keep the old data and believe you restored.&lt;/p&gt;

&lt;p&gt;I wrote the steps down and ran them. To be sure it really restored something, I created a folder with a marker in Grafana before the backup. After the restore it was there, and all 7 targets were up.&lt;/p&gt;

&lt;p&gt;A backup you haven't restored is a hope, not a backup.&lt;/p&gt;

&lt;h2&gt;
  
  
  "Ansible 2.9+"
&lt;/h2&gt;

&lt;p&gt;The README said Ansible 2.9 or newer. I had never run it on 2.9, I just wrote the number.&lt;/p&gt;

&lt;p&gt;I tried. On 2.9 the syntax check passes, then the run fails on the target with &lt;code&gt;ModuleNotFoundError&lt;/code&gt;, a known clash with Python 3.12. From 2.12 to the latest 2.21 everything worked, and the second run gave &lt;code&gt;changed=0&lt;/code&gt;. The README now says 2.12+.&lt;/p&gt;

&lt;h2&gt;
  
  
  Everything else
&lt;/h2&gt;

&lt;p&gt;The other 21 are smaller: a compose block in the wrong place, a stale link, flags in an example config that no code reads, Alertmanager needing &lt;code&gt;restart&lt;/code&gt; instead of &lt;code&gt;up -d&lt;/code&gt;. All fixed in 15 pull requests, #26 to #40.&lt;/p&gt;

&lt;p&gt;What I did not check: a real x86_64 machine (I used emulation), Debian and RHEL, load.&lt;/p&gt;

&lt;p&gt;A check I ran by hand once is a snapshot. The next version bump will break something again, so the next step is a CI smoke test: compose up, targets up, Ansible twice with &lt;code&gt;changed=0&lt;/code&gt;. Docs should run like code.&lt;/p&gt;

&lt;p&gt;How do you check that your install docs still work: CI, a manual run before a release, or you find out from an issue?&lt;/p&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/Airat71/monitoring-stack" rel="noopener noreferrer"&gt;github.com/Airat71/monitoring-stack&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>monitoring</category>
      <category>prometheus</category>
      <category>documentation</category>
    </item>
    <item>
      <title>See who's attacking your server in Grafana — fail2ban metrics with Prometheus</title>
      <dc:creator>Airat Iunusov</dc:creator>
      <pubDate>Sat, 03 Oct 2026 14:45:12 +0000</pubDate>
      <link>https://dev.to/airat71/see-whos-attacking-your-server-in-grafana-fail2ban-metrics-with-prometheus-2d7a</link>
      <guid>https://dev.to/airat71/see-whos-attacking-your-server-in-grafana-fail2ban-metrics-with-prometheus-2d7a</guid>
      <description>&lt;p&gt;Most people set up fail2ban and forget about it. It runs and bans IPs, but you only know what it does if you SSH in and read the logs.&lt;/p&gt;

&lt;p&gt;This post adds fail2ban metrics to a Prometheus stack and shows them in Grafana, on the same board as CPU, memory and disk.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F34r4t6il4w3mw51tipqv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F34r4t6il4w3mw51tipqv.png" alt="fail2ban dashboard in Grafana" width="800" height="441"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  What you will see
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Current bans and failed attempts per jail&lt;/li&gt;
&lt;li&gt;Total bans over time&lt;/li&gt;
&lt;li&gt;Which jail fires most: SSH, Nginx or recidive&lt;/li&gt;
&lt;li&gt;Whether the exporter can still reach fail2ban at all&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The exporter gives counts and jail names. It does not put banned IP addresses into metrics, so you see how much is happening, not who.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: check the socket
&lt;/h2&gt;

&lt;p&gt;The exporter reads the fail2ban control socket, not the log files.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;&lt;span class="nb"&gt;test&lt;/span&gt; &lt;span class="nt"&gt;-S&lt;/span&gt; /var/run/fail2ban/fail2ban.sock &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;ls&lt;/span&gt; &lt;span class="nt"&gt;-l&lt;/span&gt; /var/run/fail2ban/fail2ban.sock
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Stop here if that path is missing or is a directory. Expected mode is owner-only, for example &lt;code&gt;srw------- root root&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: add the exporter
&lt;/h2&gt;

&lt;p&gt;Add this block to the &lt;code&gt;services:&lt;/code&gt; section of your &lt;code&gt;docker-compose.yml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="na"&gt;fail2ban-exporter&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
  &lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;registry.gitlab.com/hctrdev/fail2ban-prometheus-exporter:0.10.3&lt;/span&gt;
  &lt;span class="na"&gt;container_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;fail2ban-exporter&lt;/span&gt;
  &lt;span class="na"&gt;restart&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;unless-stopped&lt;/span&gt;
  &lt;span class="na"&gt;ports&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1:9191:9191"&lt;/span&gt;
  &lt;span class="na"&gt;volumes&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;/var/run/fail2ban:/var/run/fail2ban:ro&lt;/span&gt;
  &lt;span class="na"&gt;read_only&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
  &lt;span class="na"&gt;security_opt&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;no-new-privileges:true&lt;/span&gt;
  &lt;span class="na"&gt;networks&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="s"&gt;monitoring-net&lt;/span&gt;
  &lt;span class="na"&gt;deploy&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;resources&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
      &lt;span class="na"&gt;limits&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;memory&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;32M&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Two things I learned the hard way.&lt;/p&gt;

&lt;p&gt;Mount the directory, not the socket file. If the file is missing when the container is created, Docker creates a directory at that path and fail2ban can no longer bind its socket.&lt;/p&gt;

&lt;p&gt;This image runs as uid 0, so the socket stays owner-only and I limit the container instead: read-only mount, &lt;code&gt;read_only&lt;/code&gt;, &lt;code&gt;no-new-privileges&lt;/code&gt;, port on localhost.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 3: add a scrape job to Prometheus
&lt;/h2&gt;

&lt;p&gt;In &lt;code&gt;prometheus.yml&lt;/code&gt;:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;job_name&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;fail2ban'&lt;/span&gt;
  &lt;span class="na"&gt;static_configs&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;targets&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="pi"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;fail2ban-exporter:9191'&lt;/span&gt;&lt;span class="pi"&gt;]&lt;/span&gt;
      &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
        &lt;span class="na"&gt;instance&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s1"&gt;'&lt;/span&gt;&lt;span class="s"&gt;your-server'&lt;/span&gt;
  &lt;span class="na"&gt;scrape_interval&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;30s&lt;/span&gt;
  &lt;span class="na"&gt;scrape_timeout&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;10s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Prometheus and the exporter must be on the same Compose network. Reload Prometheus, then check:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;docker compose up &lt;span class="nt"&gt;-d&lt;/span&gt; fail2ban-exporter
curl &lt;span class="nt"&gt;-s&lt;/span&gt; http://127.0.0.1:9191/metrics | &lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="s1"&gt;'^f2b_up'&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;f2b_up&lt;/code&gt; must be &lt;code&gt;1&lt;/code&gt;. The Prometheus target can show UP while &lt;code&gt;f2b_up&lt;/code&gt; is &lt;code&gt;0&lt;/code&gt;: the process is serving metrics but cannot use the socket.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 4: add the alerts
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight yaml"&gt;&lt;code&gt;&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Fail2banHighAttackRate&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;increase(f2b_jail_banned_total[5m]) &amp;gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;High&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;fail2ban&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;ban&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;rate&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;on&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.instance&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;

&lt;span class="pi"&gt;-&lt;/span&gt; &lt;span class="na"&gt;alert&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;Fail2banSocketDown&lt;/span&gt;
  &lt;span class="na"&gt;expr&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;f2b_up == &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
  &lt;span class="na"&gt;for&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;5m&lt;/span&gt;
  &lt;span class="na"&gt;labels&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;severity&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s"&gt;warning&lt;/span&gt;
  &lt;span class="na"&gt;annotations&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt;
    &lt;span class="na"&gt;summary&lt;/span&gt;&lt;span class="pi"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fail2ban&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;exporter&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;cannot&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;read&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;the&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;socket&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;on&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.instance&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The first one fires when more than 10 IPs are banned in 5 minutes. The second one covers the case above: the exporter is up but blind.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 5: import the dashboard
&lt;/h2&gt;

&lt;p&gt;Download &lt;code&gt;fail2ban.json&lt;/code&gt; from &lt;code&gt;grafana-dashboards/json/&lt;/code&gt; in the repo and import it in Grafana via Dashboards, Import. Select the &lt;code&gt;fail2ban&lt;/code&gt; job in the &lt;code&gt;$job&lt;/code&gt; dropdown.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why bother
&lt;/h2&gt;

&lt;p&gt;The first time I saw a ban spike in Grafana, 40 IPs banned in under 2 minutes at 3am, I realized I had had no view of this before. It was happening, I just could not see it.&lt;/p&gt;

&lt;p&gt;One gotcha: by default systemd removes &lt;code&gt;/var/run/fail2ban&lt;/code&gt; when fail2ban stops and creates a new directory on start. A container that mounted the old one keeps a deleted directory, and &lt;code&gt;f2b_up&lt;/code&gt; stays 0. A systemd drop-in with &lt;code&gt;RuntimeDirectoryPreserve=yes&lt;/code&gt; fixes it. I checked it on Ubuntu 24.04 (arm64) with fail2ban 1.0.2.&lt;/p&gt;

&lt;p&gt;What I did not check: other distributions and other fail2ban versions.&lt;/p&gt;

&lt;p&gt;Do you export fail2ban metrics, or do you still read the logs?&lt;/p&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/Airat71/monitoring-stack" rel="noopener noreferrer"&gt;github.com/Airat71/monitoring-stack&lt;/a&gt; (guide: &lt;code&gt;docs/FAIL2BAN_ENHANCED.md&lt;/code&gt;)&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Disclosure: I used an AI assistant to run the repetitive checks and to help draft this text. The findings, the decisions and the fixes are mine.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>devops</category>
      <category>prometheus</category>
      <category>grafana</category>
      <category>linux</category>
    </item>
  </channel>
</rss>
