Christoph Dieck

Posted on Sep 14 Originally published at slashgordon.link

Monitoring, or 'Why is the internet constantly asking for my .php files?

#homelab #docker #monitoring #grafana

A Grafana dashboard sounds like an enterprise thing, far too much hassle for a private homelab. I used to think so too.

Terms like Prometheus, Exporter, Loki, and Promtail sound like a lot of overhead and a steep learning curve. But I started playing around with it for fun, and thanks to Docker the whole thing was up and running faster than you can spell "monitoring". This is how I ended up monitoring my setup and turning away bots at the front door.

What do I actually want to know?

You might remember my setup from the article about Cloudflare Tunnels and Docker Stacks. I want visibility into my Nginx instances, the Proxmox host, the Ubuntu server (including SSH logs), and my Docker containers.

That's a fair number of endpoints, but there's a matching exporter for almost all of it:

pve-exporter fetches metrics directly from the Proxmox API.
node-exporter is the classic for Linux host metrics (CPU, RAM, disk).
cAdvisor looks inside Docker containers and cgroups.
promtail is the log vacuum that ships everything to Loki.

The backend is Loki for logs and Prometheus for time-series metrics. It looks like this:

graph LR
    classDef grafana fill:#F47A20,stroke:#fff,stroke-width:2px,color:#fff;
    classDef prometheus fill:#E6522C,stroke:#fff,stroke-width:2px,color:#fff;
    classDef loki fill:#000,stroke:#fff,stroke-width:2px,color:#fff;
    classDef exporter fill:#00a3cc,stroke:#fff,stroke-width:2px,color:#fff;
    classDef source fill:#444,stroke:#fff,stroke-width:1px,color:#fff,stroke-dasharray: 5 5;

    subgraph DataSources ["🔍 Data Sources"]
        LogFiles["Logfiles (Journal, Fail2ban, Docker Socket)"]:::source
        HostOS["Host OS (/proc, /sys, rootfs)"]:::source
        DockerEngine["Docker Engine (Containers, cgroups)"]:::source
        ProxmoxAPI["Proxmox VE (API)"]:::source
    end

    subgraph Agents ["🤖 Agents & Exporters"]
        Promtail["Promtail"]:::exporter
        NodeExporter["Node-Exporter"]:::exporter
        cAdvisor["cAdvisor"]:::exporter
        PVEExporter["PVE-Exporter"]:::exporter
    end

    subgraph Databases ["💾 Storage Databases"]
        Loki["Loki (Log-Aggregation)"]:::loki
        Prometheus["Prometheus (Time-Series Metrics)"]:::prometheus
    end

    Grafana["Grafana (Dashboards)"]:::grafana

    LogFiles -.-> Promtail
    Promtail --> Loki
    HostOS -.-> NodeExporter
    DockerEngine -.-> cAdvisor
    ProxmoxAPI -.-> PVEExporter

    Prometheus --> NodeExporter
    Prometheus --> cAdvisor
    Prometheus --> PVEExporter

    Grafana --> Loki
    Grafana --> Prometheus

Under Docker the configuration is mostly a copy-paste job. Only the Proxmox exporter needs manual work, since you have to create an API token with the right permissions in the PVE interface.

Dashboards

At grafana.com/dashboards there's a large selection of community dashboards you can import by ID. My favorites:

Node Exporter Full: everything about the host.
Nginx Web Analytics: who's coming from where?
cAdvisor Exporter: what are my containers eating?
SSH Logs: who is trying to log in via SSH?
CrowdSec Metrics: who's already been kicked out?

Provisioning runs from YAML files, so everything comes back instantly after a fresh install. The details are in my GitHub repo.

First discoveries: hello, botnet

As soon as the first Nginx logs showed up in Grafana, there was a lot more traffic than I expected. A second look made it clear those weren't readers, they were bots.

I was getting hammered with 404s for paths like /tool.php, /admin-footer.php, or /abc.php. My sites are static HTML built with Astro, so there is nothing here for a PHP exploit to find, but the bots keep trying every second anyway.

Log excerpt (anonymized):
request for /tool.php with HTTP status: 404 from 172.22.0.8 located in IT
request for /jga.php with HTTP status: 404 from 172.22.0.8 located in IT

The fix: Traefik and CrowdSec

I thought Cloudflare's free plan would catch all of this. It doesn't, plenty of these scans still get through. And since Traefik sits behind the Cloudflare proxy, it normally only sees the Cloudflare IP, so a ban would block the proxy itself.

The answer is CrowdSec, the modern successor to fail2ban, plus a Traefik plugin. The key is the label forwardedHeadersCustomName=CF-Connecting-IP, which tells the CrowdSec plugin to read the real visitor IP from the Cloudflare header.

Here is my docker-compose.yml snippet for the two of them:

version: "3.8"

services:
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    restart: unless-stopped
    # Entrypoint handles the bouncer key from secrets
    entrypoint: ["/bin/sh", "-c", "export BOUNCER_KEY_traefik=$(cat /run/secrets/crowdsec_bouncer_key) && exec /docker_start.sh"]
    environment:
      COLLECTIONS: "crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/base-http-scenarios"
    volumes:
      - crowdsec-config:/etc/crowdsec
      - crowdsec-data:/var/lib/crowdsec/data
      - traefik-access-logs:/var/log/traefik:ro
    networks:
      - apps
      - monitoring_net
    secrets:
      - crowdsec_bouncer_key

  traefik:
    image: traefik:v3.6.6
    container_name: traefik
    restart: unless-stopped
    depends_on:
      - crowdsec
    command:
      - "--accesslog=true"
      - "--accesslog.filepath=/var/log/traefik/access.log"
      - "--experimental.plugins.crowdsec-bouncer.modulename=github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      - "--experimental.plugins.crowdsec-bouncer.version=v1.4.1"
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.crowdseclapihost=crowdsec:8080"
      - "traefik.http.middlewares.crowdsec.plugin.crowdsec-bouncer.forwardedHeadersCustomName=CF-Connecting-IP"
    networks:
      - cloudflare
      - apps

The middleware in action

From here I just add the middleware label to every web service (portfolio, blog, and so on). Every request gets checked against the CrowdSec database, and if an IP is flagged as malicious, Traefik blocks it immediately.

blog-astro-web-prd:
  image: nginx:alpine
  labels:
    - "traefik.enable=true"
    - "traefik.http.routers.blog-astro-prd.rule=Host(`www.slashgordon.link`)"
    - "traefik.http.routers.blog-astro-prd.middlewares=crowdsec@docker"

Is it worth the effort?

For me, yes. It's reassuring to open Grafana and watch the banned-IPs count climb while the CPU load of my 35-watt roommate stays flat. The dashboard also makes the effect obvious: as soon as a bot starts scanning my .php paths, it's blocked after a few attempts.

This post was originally published on www.slashgordon.link.

DEV Community