Nagios + Grafana on Self-Hosted Hardware: A Real Monitoring Stack Without the Cloud Tax

TL;DR: The failure mode that actually bites self-hosted operators isn't a dramatic crash with error messages everywhere — it's a silent service exit at 2am that nobody notices until the automation pipeline produces zero output at 9am. A container stops responding, a cron job silently s

📖 Reading time: ~24 min

What's in this article

1. The Problem: Flying Blind on Your Own Hardware
2. Architecture Decision: Why Both Tools Instead of One
3. Deploying Nagios Core in Docker
4. Deploying Prometheus and Grafana in Docker
5. Service Checks That Actually Matter for a Self-Hosted AI Stack
6. Notification Routing and Avoiding Alert Fatigue
7. Gotchas That Cost Real Time

---

The Problem: Flying Blind on Your Own Hardware

The failure mode that actually bites self-hosted operators isn't a dramatic crash with error messages everywhere — it's a silent service exit at 2am that nobody notices until the automation pipeline produces zero output at 9am. A container stops responding, a cron job silently skips, a disk fills to 100% and writes start failing quietly. No alert fires because there's no alerting. No timeline exists because there's no metrics history. You're left doing forensics on stale logs trying to reconstruct what happened and when. That's the specific problem this guide solves.

Cloud monitoring tools don't fit this situation for two compounding reasons. First, cost: Datadog, New Relic, and similar platforms price per host or per metric volume. A homelab or small self-hosted stack with eight to twelve machines gets expensive fast, and the pricing tiers are designed for teams with budgets, not solo operators paying out of pocket. Second, and more important for the threat model: these tools work by shipping your internal metrics — service names, CPU patterns, network topology, error strings — to someone else's infrastructure. If you're running services on your own hardware specifically to avoid that kind of data exposure, sending detailed telemetry about your internal systems to a SaaS vendor undermines the whole premise.

A minimal but real monitoring stack needs three layers that most incomplete setups are missing at least one of. Host-level checks cover the basics that will kill you if ignored: CPU sustained above threshold, RAM pressure causing swap, disk partitions approaching full, load average spiking. Service-level checks go further — they verify that the thing you care about is actually responding, not just that the process is listed in ps. A Docker container can show as Up 3 days while the application inside it has deadlocked and stopped accepting connections. Visual dashboards handle the third failure mode: slow degradation. Disk growing at 2GB/day, RAM usage climbing 5% weekly — these don't trigger threshold alerts until it's already an emergency, but they're obvious on a time-series graph if you look.

The stack covered here is Nagios Core for check execution and alerting, Prometheus for metrics collection, and Grafana for dashboards — all running in Docker on the same LAN segment as the workloads they monitor. That last point matters: the monitoring stack has no dependency on external connectivity. If your internet goes down, you still get alerts on the local network. Nagios handles the "is this service up right now?" question with active checks and notification routing. Prometheus handles the "what has this looked like over the past 30 days?" question with time-series storage. Grafana sits in front of Prometheus and makes that history readable without writing PromQL from scratch every time. These tools have been around long enough that their rough edges are documented, their failure modes are known, and their Docker images are stable.

Architecture Decision: Why Both Tools Instead of One

The tempting shortcut is picking one tool and bending it to cover both jobs. Grafana has alerting. Prometheus has threshold rules. Nagios has a built-in web dashboard. Every one of those "just use this one thing" paths leads to a system that half-works in two directions instead of working well in one. The clean split is this: Nagios owns the pager, Grafana owns the screen. Once you commit to that boundary, the entire stack stops fighting itself.

Nagios Core does threshold-based alerting with a notification pipeline that's been production-hardened for decades. You define a check, set warning and critical thresholds, wire up a contact group, and when a disk hits 90% you get paged — via email, a PagerDuty webhook, or a Slack incoming hook. That pipeline is synchronous, stateful, and has built-in escalation logic. What Nagios cannot do is show you what CPU utilization looked like at 3am last Tuesday, or whether your container memory has been creeping up 200MB/week for the last month. It has no time-series store. Asking it to draw trends is like asking a smoke detector to show you a temperature graph.

That gap is exactly what the Prometheus + Grafana layer fills. On my 32GB workstation running several Ollama model-serving processes, the GPU utilization dashboard in Grafana is the only thing that's caught a slow VRAM leak before it hard-crashed the inference process. The symptom wasn't a threshold breach — VRAM never hit 100% in a single scrape — it was a 48-hour upward slope that only showed up when you looked at the sparkline. Nagios would have slept through that entirely. Grafana + nvidia_smi_exporter scraped every 30 seconds made it visible in under a day.

The overlap trap catches most people at the Grafana alerting step. Grafana can absolutely fire alerts. Prometheus has ALERT rules. If you turn both on alongside Nagios, you end up with three systems that can all page you for the same event — and worse, you'll find yourself tuning suppression rules across three different UIs to stop the duplication. The maintenance cost compounds fast. The rule is: one pager, one dashboard. Silence Grafana's alerting entirely. Do not configure Alertmanager to send pages. Let Nagios own all notification routing, period. This sounds like a constraint but it's actually a relief.

On resource cost: Nagios Core is almost free at runtime. Expect under 100MB RSS even with a few hundred active checks. Prometheus is a different story. With a 15-day retention window, scraping 5 hosts every 15 seconds at moderate metric cardinality (node_exporter default is around 800 metrics per host), you'll accumulate roughly 2–4GB of TSDB data per day depending on how many containers and custom exporters you add. Before you deploy, size your --storage.tsdb.retention.time and make sure the target volume can absorb it. On a home lab with a spinning disk, that's manageable. On a VPS with expensive block storage, it becomes a cost decision. The flag to control it in your compose file looks like this:

command:
  - '--config.file=/etc/prometheus/prometheus.yml'
  - '--storage.tsdb.path=/prometheus'
  - '--storage.tsdb.retention.time=15d'   # tune before first deploy
  - '--storage.tsdb.retention.size=20GB'  # hard cap as a safety net

Set both flags. The time-based retention is the primary control; the size cap is the circuit breaker if your cardinality explodes after adding a new exporter. Without the size cap, a misconfigured exporter emitting high-cardinality labels (per-request URLs are the classic offender) can fill a disk inside 48 hours on an otherwise healthy setup.

Deploying Nagios Core in Docker

The jasonrivers/nagios image is the community-maintained option that's actually kept up — the official Nagios dockerfiles have gone stale. Pin to a specific tag. 4.4.14 is stable as of this writing. The reason to pin isn't pedantry: Nagios config syntax has genuinely shifted between minor versions, particularly around the use inheritance directive and host template parsing. Running latest means a surprise rebuild can silently break your config validation, and you won't know until you restart the container and nagios refuses to start.

Here's a minimal docker-compose.yml that mounts your config and state directories separately — this matters because you want to edit config files outside the container lifecycle, and you don't want a container rebuild wiping your status history:

services:
  nagios:
    image: jasonrivers/nagios:4.4.14
    container_name: nagios
    restart: unless-stopped
    ports:
      # Bind only to your LAN interface — never 0.0.0.0 on a monitoring host
      - "192.168.1.10:8080:80"
    volumes:
      - ./nagios/etc:/opt/nagios/etc      # all config lives here
      - ./nagios/var:/opt/nagios/var      # status.dat, logs, retention
    environment:
      - NAGIOSADMIN_USER=nagiosadmin
      - NAGIOSADMIN_PASS=changeme_before_first_run

The port binding 192.168.1.10:8080:80 is doing real work. If you bind to 0.0.0.0, the Nagios web UI is reachable from anywhere the host is reachable, which on a typical home lab includes WAN if you've got port forwarding open for other services. Bind to your actual LAN IP and that attack surface disappears. Your first run goal is simple: get the UI green on the default localhost host before touching anything else.

Once the container is up, add your first real host. Drop a file at ./nagios/etc/objects/hosts.cfg — the container's default nagios.cfg includes cfg_dir=/opt/nagios/etc/objects so any .cfg file there gets picked up:

define host {
    host_name               fileserver
    alias                   Main File Server
    address                 192.168.1.20
    check_command           check-host-alive
    max_check_attempts      3
    check_period            24x7
    notification_period     24x7
    contacts                nagiosadmin
}

Validate before restarting — this catches syntax errors without a full restart cycle:

docker exec nagios /opt/nagios/bin/nagios -v /opt/nagios/etc/nagios.cfg

If validation passes, restart the container and confirm the host shows green in the web UI. Don't pile on service checks until you've confirmed ICMP reach works. A host that shows as DOWN before you've added any services tells you immediately whether it's a network or config problem.

For real host metrics — disk, CPU, running processes — you need NRPE on each monitored machine. The nagios-nrpe-server package on Ubuntu/Debian installs the daemon and a default config at /etc/nagios/nrpe.cfg. The critical setting that most setups get wrong initially is allowed_hosts. It must contain the IP your Nagios container uses when making outbound connections — which is the Docker host's LAN IP if you're using bridge networking, not the container's internal IP:

# /etc/nagios/nrpe.cfg on the monitored host
allowed_hosts=127.0.0.1,192.168.1.10   # 192.168.1.10 = your Nagios Docker host

# These ship by default — verify they're present and paths are correct for your distro
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

On the Nagios container side, you need a check_nrpe command definition. The jasonrivers/nagios image ships with the NRPE plugin already installed, so you only need the command object in your config:

# ./nagios/etc/objects/commands.cfg — append this block
define command {
    command_name    check_nrpe
    # -H is the target host, -c is the command name defined in nrpe.cfg
    command_line    /opt/nagios/libexec/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Then wire it into a service definition for your host:

define service {
    host_name               fileserver
    service_description     Disk /
    check_command           check_nrpe!check_disk
    check_interval          5
    max_check_attempts      3
    check_period            24x7
    notification_period     24x7
    contacts                nagiosadmin
}

A common failure mode here: NRPE returns "Connection refused" even with correct allowed_hosts. Check whether nagios-nrpe-server is actually listening — ss -tlnp | grep 5666 on the monitored host. Some Ubuntu installs require systemctl enable --now nagios-nrpe-server explicitly. The other gotcha is firewall rules: if ufw is active, you need ufw allow from 192.168.1.10 to any port 5666 on the monitored host before NRPE will respond.

Deploying Prometheus and Grafana in Docker

The Scrape Chain Before You Touch Compose

The mental model that makes this whole stack click: Prometheus pulls, it never receives pushes. Every host you want to monitor needs something listening on a port that Prometheus can scrape. For bare-metal or VM system metrics, that's node_exporter on port 9100. For per-container metrics on your Docker host, that's cadvisor on port 8080 — except if Nagios is already bound to 8080 (a common default for its web UI), you'll hit a silent conflict. Run cadvisor on 8081 instead and adjust your prometheus.yml accordingly. The conflict won't crash anything; Docker will just refuse to start cadvisor, and you'll have a gap in your metrics you won't notice until you wonder why all your container panels are empty.

Node exporter runs best as a systemd service on bare-metal hosts rather than a Docker container, because getting accurate host-level metrics from inside a container requires --pid=host --net=host --privileged — at which point you've basically given it host access anyway. The systemd approach is cleaner:

# Download the current release (check github.com/prometheus/node_exporter/releases for latest)
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvf node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/

# /etc/systemd/system/node_exporter.service
[Unit]
Description=Prometheus Node Exporter
After=network.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

Cadvisor on the Docker host is a different story — it needs access to the Docker socket and cgroup filesystem, so running it as a container with the right mounts is the standard approach. Pin the version; the latest tag on cadvisor has historically had breaking changes between minor versions:

cadvisor:
  image: gcr.io/cadvisor/cadvisor:v0.49.1
  container_name: cadvisor
  privileged: true
  ports:
    - "8081:8080"   # left side changed to avoid Nagios conflict
  volumes:
    - /:/rootfs:ro
    - /var/run:/var/run:ro
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro
    - /dev/disk/:/dev/disk:ro
  restart: unless-stopped

prometheus.yml and the Retention Flag You Shouldn't Skip

The default retention in Prometheus is 15 days, which sounds fine until you realize the default storage path (/prometheus inside the container) will eat through disk silently. On a host with a small root partition, this shows up as a full disk with no obvious culprit. Always pass --storage.tsdb.retention.time=15d explicitly in your Docker command or compose file — not because it changes the default, but because it documents your intent and makes the behavior predictable when you later tune it. A minimal but complete prometheus.yml:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets:
          - 'host1.lan:9100'
          - 'host2.lan:9100'
          # add every host running node_exporter here

  - job_name: 'cadvisor'
    static_configs:
      - targets:
          - 'localhost:8081'   # matches the remapped port above

And the Prometheus container block in compose, with the retention flag explicit and storage on a named volume so data survives container rebuilds:

prometheus:
  image: prom/prometheus:v2.53.0
  container_name: prometheus
  command:
    - '--config.file=/etc/prometheus/prometheus.yml'
    - '--storage.tsdb.path=/prometheus'
    - '--storage.tsdb.retention.time=15d'   # explicit, not assumed
    - '--web.enable-lifecycle'               # lets you reload config via POST /-/reload
  volumes:
    - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    - prometheus_data:/prometheus
  ports:
    - "9090:9090"
  restart: unless-stopped

Grafana: Provisioning Over Manual Config

If you configure the Prometheus data source manually through the Grafana UI, the next time you rebuild the container it's gone. Provisioning via YAML solves this — Grafana reads datasource definitions from a mounted directory at startup. The setup requires two pieces: the compose block and a provisioning file on disk.

grafana:
  image: grafana/grafana:11.1.0
  container_name: grafana
  environment:
    - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}  # set in .env, never hardcode
    - GF_USERS_ALLOW_SIGN_UP=false
  volumes:
    - grafana_data:/var/lib/grafana
    - ./grafana/provisioning:/etc/grafana/provisioning:ro
  ports:
    - "3000:3000"
  depends_on:
    - prometheus
  restart: unless-stopped

# grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090   # container name resolves via Docker DNS
    isDefault: true
    editable: false               # prevents accidental UI changes from persisting

The editable: false flag is worth setting if you want the provisioned config to be the source of truth. Without it, someone can modify the datasource in the UI, and the next restart will silently overwrite their changes with the provisioned values — which is confusing to debug. Make the behavior explicit either way.

Dashboards 1860 and 14282: Useful Starting Points, Not Blind Trusts

Dashboard ID 1860 (Node Exporter Full) and 14282 (Docker and system monitoring via cadvisor) are real, published dashboards on grafana.com. Both are worth importing as a starting point — they save hours of panel configuration. Import them via Grafana's UI at Dashboards → Import → Enter ID, or provision them as JSON files. The honest caveat: both dashboards include panels that query metrics your specific exporter version or configuration may not expose. A panel showing "No data" isn't broken Grafana — it means the metric the panel queries (node_filesystem_avail_bytes with a specific label, for instance) doesn't exist in your scrape. Audit the panel queries before you put these dashboards on a wall display or alert on them. Delete panels that consistently return no data rather than leaving a dashboard full of empty graphs, which trains you to ignore gaps.

• 1860 common gap: Some network interface panels use label matchers that don't match non-standard interface names (like enp3s0 vs eth0). Edit the panel query's device label filter to match your actual interface names.
• 14282 common gap: Container name labels in cadvisor changed format across versions. If you're on cadvisor v0.47+, the name label on container metrics dropped the leading slash — old dashboard queries using =~"/.*" may not match.
• Both dashboards: Check which Grafana version they target. A dashboard built for Grafana 9 using the old Prometheus query editor may render differently in Grafana 11, particularly around panel transformations.

Service Checks That Actually Matter for a Self-Hosted AI Stack

Most Nagios setups default to checking whether a port is open and calling it done. For a self-hosted AI stack, that misses every failure mode that actually bites you — the Ollama process that's listening but has unloaded all models due to memory pressure, the n8n container that restarts every 90 seconds but always passes a "is it running?" check, the model storage volume that hits 95% because you pulled a 70B weight without thinking.

Checking Ollama Properly: /api/tags, Not Just Port 11434

A TCP port check against 11434 will return OK even if Ollama is mid-crash or stuck in an unresponsive state after an OOM kill. The /api/tags endpoint is the right target — it forces a real HTTP round-trip through the Ollama HTTP layer and returns a 200 with a JSON body listing loaded models. If that returns a 200, the server is actually alive. Use check_http with the -u and -e flags:

# In your Nagios commands.cfg or services config
define command {
    command_name  check_ollama_api
    command_line  $USER1$/check_http -H $HOSTADDRESS$ -p 11434 \
                  -u /api/tags -e "200 OK" \
                  -w 5 -c 10
    # -w/-c are response time thresholds in seconds
    # Ollama can be slow to respond if a model is mid-load
}

define service {
    use                   generic-service
    host_name             ai-workstation
    service_description   Ollama API Health
    check_command         check_ollama_api
}

Set your warning threshold at 5 seconds and critical at 10. A healthy Ollama responding to /api/tags with no models loading takes under 200ms. If it's crossing 5s, something is wrong with the model server even if the port is technically open. On my 32GB box, slow responses to that endpoint have reliably preceded an OOM event by a few minutes — the server was thrashing trying to keep a model resident.

Disk Checks Against the Model Storage Mount Specifically

Ollama model weights live in ~/.ollama/models by default, and a single model file can run anywhere from 4GB (a quantized 7B) to 70GB+ for a full-precision 70B. The root partition check that ships with most Nagios templates is useless here — model storage is almost always on a separate mount or at minimum a different logical volume. Check the actual mount point where model files land:

# Using check_disk against a dedicated model storage mount
define command {
    command_name  check_model_storage
    command_line  $USER1$/check_disk -w 20% -c 10% \
                  -p /mnt/models
    # -w 20% means warn when 20% FREE remains (i.e., 80% used)
    # Nagios check_disk uses free space, not used — easy to get backwards
}

define service {
    use                   generic-service
    host_name             ai-workstation
    service_description   Model Storage Disk
    check_command         check_model_storage
}

Note the inverted logic: check_disk warns when free space drops below the threshold, so -w 20% triggers the warning when you're at 80% used, and -c 10% fires critical at 90% used. If you write it as -w 80% thinking "warn at 80%", you'll fire warnings constantly because 80% free space almost never exists on a volume full of model weights. Get this backwards once and you'll spend 20 minutes debugging why your disk check is always critical.

Docker Container State via NRPE: Catching Silent Exits

Docker's built-in healthcheck and Nagios's check_procs both miss the same failure: a container that exited cleanly with code 0 and had no restart policy. The Docker daemon reports the engine as healthy, the process table shows no zombie, and everything looks fine until you notice n8n hasn't processed a workflow in 3 hours. A custom NRPE script catches this:

#!/bin/bash
# /usr/local/lib/nagios/plugins/check_docker_container
# Run via NRPE on the host — requires docker group membership for the nagios user

CONTAINER=$1

if [ -z "$CONTAINER" ]; then
    echo "UNKNOWN: No container name specified"
    exit 3
fi

RUNNING=$(docker inspect --format='{{.State.Running}}' "$CONTAINER" 2>/dev/null)
EXIT_CODE=$(docker inspect --format='{{.State.ExitCode}}' "$CONTAINER" 2>/dev/null)

if [ $? -ne 0 ]; then
    echo "CRITICAL: Container $CONTAINER not found"
    exit 2
fi

if [ "$RUNNING" = "true" ]; then
    echo "OK: $CONTAINER is running"
    exit 0
else
    echo "CRITICAL: $CONTAINER is not running (exit code: $EXIT_CODE)"
    exit 2
fi

# In nrpe.cfg on the monitored host
command[check_n8n_container]=/usr/local/lib/nagios/plugins/check_docker_container n8n
command[check_ollama_container]=/usr/local/lib/nagios/plugins/check_docker_container ollama

Add the nagios user to the docker group on the host, or this script will fail silently with a permission error that $? returns as non-zero but the output is empty. That produces CRITICAL: Container n8n not found even when the container is fine — a confusing false positive that wastes time.

n8n Crash-Loop Detection: Uptime Window Check

The /healthz endpoint on n8n (exposed on port 5678 by default) returns a 200 if the process is responsive. That's necessary but not sufficient. A container that crash-loops every 90 seconds with a --restart=always policy will pass the healthcheck right after each restart — it's technically healthy for 60 of those 90 seconds. A separate check against container uptime catches this:

#!/bin/bash
# /usr/local/lib/nagios/plugins/check_container_uptime
# Warns if a container has been running for less than N seconds — 
# designed to catch restart loops, not replace the running-state check

CONTAINER=$1
MIN_UPTIME=${2:-300}  # default: warn if running less than 5 minutes

STARTED=$(docker inspect --format='{{.State.StartedAt}}' "$CONTAINER" 2>/dev/null)

if [ $? -ne 0 ]; then
    echo "UNKNOWN: Cannot inspect $CONTAINER"
    exit 3
fi

# Convert ISO8601 timestamp to epoch — requires GNU date
STARTED_EPOCH=$(date -d "$STARTED" +%s 2>/dev/null)
NOW_EPOCH=$(date +%s)
UPTIME=$((NOW_EPOCH - STARTED_EPOCH))

if [ "$UPTIME" -lt "$MIN_UPTIME" ]; then
    echo "WARNING: $CONTAINER only up for ${UPTIME}s (threshold: ${MIN_UPTIME}s) — possible restart loop"
    exit 1
fi

echo "OK: $CONTAINER up for ${UPTIME}s"
exit 0

# NRPE command entry
command[check_n8n_uptime]=/usr/local/lib/nagios/plugins/check_container_uptime n8n 300

Pair this with a 2-minute Nagios check interval and a 1-occurrence alert threshold. A single restart isn't always alarming — Docker will restart a container after an update or a host reboot. The signal you want is the check firing on consecutive runs: if n8n's uptime resets between each 2-minute polling cycle, something is actively wrong. Set max_check_attempts 3 and notification_interval 0 in the service definition so you get one alert per event, not a flood every 2 minutes during a crash loop.

Notification Routing and Avoiding Alert Fatigue

The most common failure mode in self-hosted monitoring isn't missing an alert — it's generating so many that you start unconsciously filtering them yourself. Once your brain learns to skim past Nagios emails, the whole system is broken. The fix isn't smarter alerts; it's routing the right severity to the right channel before you ever hit that point.

Set up two contacts: one for email via Postfix or an SMTP relay container, one for Slack via a shell script wrapping curl. The Slack contact should receive CRITICAL only. Email handles WARNING. This isn't just aesthetics — a Slack ping at 2am demands attention, an email at 6am gets triaged with coffee. Here's the shell script pattern for the Slack contact:

#!/bin/bash
# /usr/local/bin/notify-slack.sh
# Called by Nagios with env vars set by the contact definition
curl -s -X POST "$SLACK_WEBHOOK_URL" \
  -H 'Content-type: application/json' \
  --data "{
    \"text\": \"*[$NAGIOS_NOTIFICATIONTYPE]* $NAGIOS_HOSTNAME: $NAGIOS_SERVICESTATE — $NAGIOS_SERVICEOUTPUT\"
  }"

Wire it into the contact definition like this:

define contact {
    contact_name                    ops-slack
    service_notification_commands   notify-service-by-slack
    service_notification_options    c          ; CRITICAL only
    host_notification_options       d,u        ; DOWN and UNREACHABLE only
    service_notification_period     24x7
}

define command {
    command_name    notify-service-by-slack
    command_line    /usr/local/bin/notify-slack.sh
}

The notification_interval trap catches nearly every solo operator eventually. Nagios defaults to re-notifying every 30 minutes on a still-failing check. If you're running overnight model jobs on a GPU box and something goes sideways at 11pm, you will wake up to a dozen Slack pings by morning and start associating the channel with noise. For a single-operator setup, push that to 60 or 120 minutes on non-critical services:

define service {
    host_name                       gpu-workstation
    service_description             VRAM Usage
    check_command                   check_vram_util
    notification_interval           120   ; re-notify every 2 hours, not 30 min
    first_notification_delay        5
    notification_options            w,c,r
    contacts                        ops-email
}

Scheduled downtime is the other lever people skip until they regret it. Patching a box at midnight without setting downtime first means Nagios fires host-down and service-down notifications for every check on that host — easily 30-40 emails depending on how many services you're monitoring. The Nagios web UI handles this under System → Schedule Downtime, but for anything you're automating — cron-triggered reboots, maintenance scripts — hit the CGI endpoint directly:

# Schedule 90 minutes of downtime for host "gpu-workstation" starting now
curl -s -u nagiosadmin:yourpassword \
  "http://localhost/nagios/cgi-bin/cmd.cgi" \
  --data "cmd_typ=55&cmd_mod=2&host=gpu-workstation&com_author=cron&com_data=planned+reboot&trigger=0&start_time=$(date +'%m-%d-%Y+%H:%M:%S')&end_time=$(date -d '+90 minutes' +'%m-%d-%Y+%H:%M:%S')&fixed=1&hours=1&minutes=30&childoptions=0"

One thing worth keeping straight: silence via downtime is not the same as filtering via notification options. Downtime suppresses alerts while still running checks and recording state. A contact with service_notification_options w stripped out simply never routes WARNING notifications to that contact — the check still fires, the state still changes, Grafana still graphs it. You want both mechanisms. Downtime for planned maintenance, filtered routing for permanent severity separation. Conflating them leads to gaps where you think you're monitoring something but no channel is actually wired to receive it. For operators building broader pipelines where this notification layer feeds into webhook triggers or self-healing automation, the architecture overlaps with general workflow tooling — the Workflow Automation in 2026: n8n, Zapier, and Self-Hosted Pipelines guide covers where those systems connect.

Gotchas That Cost Real Time

The most expensive one: Nagios silently stops checking things when you reload a bad config, and the container keeps running like nothing happened. Before you ever run docker exec nagios kill -HUP 1, verify the config file first:

# Run this inside the container or via exec — never skip it
docker exec nagios /usr/local/nagios/bin/nagios -v /opt/nagios/etc/nagios.cfg

# Expected tail of good output:
# Total Warnings: 0
# Total Errors:   0
# Things look okay - No serious problems were detected during the pre-flight check

A typo in a host definition, a missing use template reference, or a circular dependency in service groups will cause the reload to fail. Nagios won't crash — it keeps running the last known good config. But if you made that config change because a host was added, that host gets zero checks, zero alerts, and you won't notice until something on it actually breaks and nobody pages you. Make nagios -v a pre-commit hook or at minimum a manual step before every reload.

Prometheus cardinality is the slower-burning problem. cAdvisor emits labels like container_label_com_docker_compose_service, image, interface, and several others per time series. On a stable set of long-running containers that's manageable. The moment you start running ephemeral containers — n8n's Execute Command nodes, one-shot ETL jobs, anything that spins up and tears down per workflow run — you accumulate unique label value combinations that never get cleaned up within the retention window. Query performance on container_cpu_usage_seconds_total starts to drag. The fix is metric_relabel_configs in your Prometheus scrape config, applied before the data lands in TSDB:

scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ['cadvisor:8080']
    metric_relabel_configs:
      # Drop per-interface network metrics you're not dashboarding
      - source_labels: [__name__]
        regex: 'container_network_(receive|transmit)_(packets|errors|dropped)_total'
        action: drop
      # Strip high-cardinality compose labels from all metrics
      - regex: 'container_label_com_docker_(compose_config_hash|compose_version|swarm.*)'
        action: labeldrop

Drop rules are evaluated before storage, so they reduce active series immediately. Don't wait until Prometheus is already slow to add these — by then you're also dealing with a bloated WAL.

Grafana's provisioning behavior bites almost everyone once. If you mount a dashboard JSON into /etc/grafana/provisioning/dashboards/ and Grafana picks it up on startup, that dashboard is now managed by the provisioning system. Edit it in the UI, save it, restart the container — your edits are gone. Grafana overwrites the dashboard from the file on startup. The options are: keep the JSON file as the single source of truth and make all edits there (export from UI, update the file, restart), or after the initial import remove the provisioning config entry so Grafana owns it going forward. What you cannot do is treat both as writeable simultaneously. Pick a lane.

NRPE firewall rules through Docker are genuinely tricky and the standard advice fails here. Running ufw allow from 192.168.1.50 to any port 5666 on the monitored host feels correct, but Docker's iptables manipulation means the Nagios container's traffic arrives on the host with the bridge network's IP, and your ufw rule may never match it. The correct test is always from inside the Nagios container, not from the host itself:

# Test from inside the Nagios container
docker exec -it nagios nc -zv 192.168.1.200 5666

# If that fails but the host-level nc works, you have a Docker FORWARD chain issue
# Check what Docker inserted:
iptables -L DOCKER -n --line-numbers
iptables -L FORWARD -n --line-numbers | grep 5666

If you're using ufw, the reliable path is adding the NRPE allow rule directly via iptables targeting the Docker bridge subnet, or switching the monitored host's NRPE binding to a specific interface and exposing the port in the compose file with explicit host binding. Testing from the Docker network namespace rather than from the bare host catches this class of bug before you spend an hour wondering why Nagios reports all NRPE checks as unreachable.

---

Disclaimer: This article is for informational purposes only. The views and opinions expressed are those of the author(s) and do not necessarily reflect the official policy or position of Sonic Rocket or its affiliates. Always consult with a certified professional before making any financial or technical decisions based on this content.

---

Originally published on techdigestor.com. Follow for more developer-focused tooling reviews and productivity guides.