DEV Community: david

SLO Burn-Rate Alerting with Prometheus: Beyond Threshold Alerts

david — Wed, 24 Jun 2026 19:03:59 +0000

Most uptime alerts look like this:

- alert: ServiceDown
  expr: probe_success == 0
  for: 2m

That fires when a service is completely down for two minutes. It won't fire when a service is responding to 95% of requests for 48 hours straight — even though that's silently consuming your entire monthly error budget.

Burn-rate alerting is a different model. Instead of alerting on current state, it alerts on how fast you're spending your error budget. A 30x burn rate means you'll exhaust your entire month of tolerance in about 50 minutes. A 6x burn rate means you have a few hours. Both warrant action — just different kinds of action.

This is the implementation running on my bare-metal k3s cluster, based directly on the multi-window multi-burn-rate approach from the Google SRE Workbook.

View the complete homelab infrastructure source on GitHub 🐙

Error Budgets, Briefly

If your SLO is 99.9% availability, your monthly error budget is the allowed downtime: 43.8 minutes per month (0.1% of 43,800 minutes).

The core insight: not all errors are the same urgency. A service that's been returning errors at 30x the normal rate for the past two hours will exhaust that 43.8-minute budget in ~50 minutes — that's a page. A service burning at 6x for the past six hours has 4 hours left — that's a ticket, handled during the shift.

Threshold alerting conflates these. Burn-rate alerting separates them.

The SLI: HTTP Probe Success Rate

Everything is built on a single Service Level Indicator: the fraction of successful HTTP probes from the Prometheus blackbox exporter.

The blackbox exporter probes each public service endpoint on a fixed interval. probe_success is 1 for a successful probe and 0 for a failure. The SLI is the average over a time window:

# kubernetes/system/monitoring/slo-rules.yml

- record: job_instance:probe_success:rate5m
  expr: avg_over_time(probe_success[5m])

- record: job_instance:probe_error:rate5m
  expr: 1 - avg_over_time(probe_success[5m])

1 - success_rate = error_rate. At 99.9% SLO, the allowed steady-state error rate is 0.001 (0.1%).

Recording Rules: Pre-Computing the Windows

Multi-window alerting needs error rates computed over multiple time windows. Prometheus can do this inline in alert expressions, but pre-computing them as recording rules keeps the alert expressions readable and reduces query load.

- name: slo.availability.windows
  interval: 1m
  rules:
    # Short windows (fast-burn detection)
    - record: job_instance:probe_success:rate1h
      expr: avg_over_time(probe_success[1h])
    - record: job_instance:probe_success:rate2h
      expr: avg_over_time(probe_success[2h])

    # Medium windows
    - record: job_instance:probe_success:rate6h
      expr: avg_over_time(probe_success[6h])
    - record: job_instance:probe_success:rate30m
      expr: avg_over_time(probe_success[30m])

    # Long windows (slow-burn detection)
    - record: job_instance:probe_success:rate24h
      expr: avg_over_time(probe_success[24h])

These evaluate every minute. The result is a set of pre-computed availability metrics across six time windows — from 30 minutes (most sensitive) to 24 hours (catches slow bleeds).

The Alert Rules

Fast Burn: Page Immediately

- alert: SLOAvailabilityFastBurn
  expr: |
    (1 - job_instance:probe_success:rate2h) > (30 * (1 - 0.999))
    and
    (1 - job_instance:probe_success:rate1h) > (30 * (1 - 0.999))
  for: 2m
  labels:
    severity: critical
    slo: availability
  annotations:
    summary: "SLO fast burn: {{ $labels.instance }}"
    description: >
      {{ $labels.instance }} error rate is burning through the monthly error budget
      at ≥30x the allowed rate. At this pace the 99.9% budget is exhausted in ~50min.
      Current 2h error rate: {{ printf "%.2f" $value }}%

The math: A 99.9% SLO means 0.1% of requests can fail. The threshold for 30x burn is 30 × 0.001 = 0.03 — a 3% error rate. If both the 2-hour window and the 1-hour window exceed 3%, this fires.

Why two windows? The short window (1h) catches fast-developing incidents. The long window (2h) provides confirmation — it prevents a single spike from paging. Both must exceed the threshold simultaneously. This dual-window check is the key difference from naive threshold alerting: a two-minute blip won't page you, but a sustained fast burn will.

Burn-rate math at 30x:

Monthly budget: 43.8 minutes
At 30x burn: 43.8 ÷ 30 = 1.46 minutes consumed per minute
Budget exhausted in: 43.8 ÷ (30 - 1) ≈ 51 minutes

51 minutes to act. Page.

Slow Burn: Create a Ticket

- alert: SLOAvailabilitySlowBurn
  expr: |
    (1 - job_instance:probe_success:rate6h) > (6 * (1 - 0.999))
    and
    (1 - job_instance:probe_success:rate30m) > (6 * (1 - 0.999))
  for: 15m
  labels:
    severity: warning
    slo: availability
  annotations:
    summary: "SLO slow burn: {{ $labels.instance }}"
    description: >
      {{ $labels.instance }} error rate is burning through the monthly error budget
      at ≥6x the allowed rate. At this pace the 99.9% budget is exhausted in ~4h.
      Current 6h error rate: {{ printf "%.2f" $value }}%

The math: 6 × 0.001 = 0.006 — a 0.6% error rate. Budget exhaustion at 6x burn: 43.8 ÷ (6 - 1) ≈ 8.8 hours. The for: 15m means it must sustain this rate for 15 minutes before firing, which filters transient dips.

6h (long) + 30m (short) windows. A slow degradation is visible over 6 hours; the 30m short window prevents false positives from stale data.

Severity: warning. This goes to a Slack channel, not a pager. Fix it during the shift.

Comparing Against Threshold Alerting

Scenario	Threshold alert (`< 99%`)	Burn-rate alert
Service down for 2 minutes	✅ Fires	✅ Fires (fast burn)
Service at 95% for 48h	❌ Fires then resolves	✅ Fires slow burn, escalates
3% error rate for 1h	❌ May not fire	✅ Fast burn fires
0.5% error rate for 6h	❌ Never fires	✅ Slow burn fires
Single 10-second blip	✅ Fires (false positive)	❌ Below `for` threshold

The pattern: burn-rate alerting catches slow degradations that threshold alerting misses, and it filters the transient blips that threshold alerting over-alerts on.

Deploying as a PrometheusRule

The rules deploy as a PrometheusRule CRD, picked up automatically by the Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: homelab-slo-alerts
  namespace: monitoring
  labels:
    prometheus: kube-prometheus
    role: alert-rules
spec:
  groups:
    - name: slo.burn-rate.page
      rules:
        - alert: SLOAvailabilityFastBurn
          # ... (see above)

The prometheus: kube-prometheus label tells the Prometheus Operator to load this rule. kubectl get prometheusrule -n monitoring should show it; kubectl get --raw /api/v1/namespaces/monitoring/pods/prometheus-kube-prometheus-prometheus-0/proxy/api/v1/rules lets you query the loaded rules directly.

What the Error Budget Dashboard Shows

The complementary Grafana dashboard (slo-dashboard.yml) renders three panels:

Availability over time — job_instance:probe_success:rate5m across all probed services
Error budget remaining — 1 - (sum(rate(probe_success[30d])) / count(probe_success)) relative to the 0.1% budget
Burn rate — current consumption rate, coloured by severity tier

The budget panel is the most useful. When it's dropping steeply, something is consuming more than the flat weekly allocation. That's a signal even before an alert fires.

Limitations

This implementation measures external availability only — HTTP probes from inside the cluster. It won't catch:

Increased latency that doesn't fail probes (need histogram SLIs for that)
Internal service-to-service degradation (need distributed tracing or internal probes)
Correctness issues — a 200 OK with wrong data doesn't fail a probe

For most homelab services — Nextcloud, Authelia, Jellyfin, Gitea — availability is the right SLI. For a production API, you'd want to add latency SLOs (P99 < 500ms) using histogram recording rules.

The same pattern applies directly to enterprise environments. If you're running Azure Load Balancer health probes or Application Gateway, the SLI is the same: probe success rate. The recording rules and alert thresholds are identical. The only difference is where the metrics come from.

I Hardened Pod securityContext and Broke 9 Containers in Production

david — Wed, 24 Jun 2026 19:03:35 +0000

kubeconform passed. kubectl --dry-run passed. The PR looked exactly like what every Kubernetes security checklist tells you to do: capabilities.drop: [ALL], runAsNonRoot: true, allowPrivilegeEscalation: false across every container that was missing a securityContext. Schema-valid, reviewed, merged.

Within minutes — because this cluster runs ArgoCD with selfHeal: true, where merge is deploy — nine containers were down. Two of them were Postgres, backing Paperless and Nextcloud. That's not a degraded non-critical service; that's an outage.

This is the failure analysis, the two wrong assumptions that caused it, the trap that bit during recovery, and the lesson for the next time anyone — including future me — is tempted to do a blanket securityContext pass across a manifest tree.

View the complete homelab infrastructure source on GitHub 🐙

The Two Wrong Assumptions

Assumption 1: capabilities.drop: [ALL] is always safe if the container doesn't need special privileges at runtime.

Wrong. It's not about what the final running process needs — it's about what the entrypoint script needs before it execs into that process. A huge number of container images follow the same pattern: start as root, chown/chmod the data directory so it's owned by an unprivileged user, then drop privileges via su-exec or setpriv before launching the actual application. That privilege-drop step itself requires CAP_CHOWN, CAP_SETUID, and CAP_SETGID — capabilities that drop: [ALL] removes before the entrypoint ever runs.

# What looked like the safe, recommended hardening:
securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]

This broke gitea, authelia, headscale, mealie, and both Postgres instances (paperless and nextcloud) — every one of them runs this exact root-then-drop-privileges pattern in its entrypoint. It also broke the Paperless and Nextcloud Redis instances — but, tellingly, not the Authelia Redis instance, because that one has an explicit command: redis-server ... override that bypasses the image's normal entrypoint script entirely. Same image, same securityContext, different outcome — because the actual code path that runs is different.

Assumption 2: runAsNonRoot: true is safe to set on any container, since "obviously" you want it not running as root.

Wrong in the opposite direction. runAsNonRoot: true doesn't change anything about how the container runs — it's an admission-time check that fails outright if the image's actual default user is root and nothing in the pod spec overrides it. vault-unseal (hashicorp/vault), the Nextcloud and Paperless Redis instances, and cloudflare-ddns (curlimages/curl) all default to root. These containers didn't crash-loop — they never started at all:

Error: container has runAsNonRoot and image will run as root

That's a CreateContainerConfigError, a clean failure with a clear message — which made it one of the easier categories to diagnose. The crash-looping containers from Assumption 1 were the harder half.

Catching It: Why "Application: Synced/Healthy" Lied

The first instinct when something looks wrong is to check ArgoCD. kubectl get application -n argocd showed Synced and Healthy. That was stale — ArgoCD's poll interval meant the Application object hadn't refreshed its view of the cluster yet, even though the new pods were already failing underneath it.

# Don't trust Application status alone during an active incident
argocd app get <name> --hard-refresh
# or:
kubectl annotate application <name> -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite

The only thing that actually told the truth was looking directly at pod status and the pod's own creationTimestamp:

kubectl get pods -n apps -o wide
kubectl get pod <name> -n apps -o jsonpath='{.metadata.creationTimestamp}{"\n"}{.status.containerStatuses[0].restartCount}'
kubectl logs <name> -n apps --previous

A pod with N restarts and a recent restart count "looking survivable" is not proof of health. Two failures in this incident — paperless-ngx and uptime-kuma — surfaced only on a slower ReplicaSet rollout and weren't caught in the first sweep immediately after merge. They were found ~30 minutes later during an extended verification pass, specifically because someone went back and checked for a clean creationTimestamp with zero restarts since — not just "fewer restarts than expected." The bar for "this is actually fixed" has to be zero restarts on the current generation, not a restart count that happens to look low.

The Logs Told the Real Story Every Time

Once you're looking at the right pod, kubectl logs on the crashing container is unambiguous:

chown: /config: Operation not permitted
su-exec: setgroups(0): Operation not permitted
setpriv: setresuid failed: Operation not permitted

Three different error message formats, same root cause: the entrypoint tried to drop privileges and couldn't, because the capability that does that had been dropped first. This is the single most useful debugging fact from the whole incident — if you see any of these three error patterns after a securityContext change, the fix is "give the capability back," not "investigate the application."

The Recovery Trap: selfHeal Undoes Manual Fixes

To restore service faster than waiting on a PR review cycle, the instinct during an active outage is to patch the live cluster directly:

kubectl patch deployment authelia -n apps --type=json \
  -p '[{"op": "remove", "path": "/spec/template/spec/containers/0/securityContext/capabilities"}]'

This works — for about as long as it takes ArgoCD's next reconciliation loop to notice the drift. With selfHeal: true, ArgoCD's entire job is to make the live cluster match Git. A manual kubectl patch that diverges from the committed manifest is drift, by definition, and gets silently reverted back to the still-broken state.

With selfHeal enabled, Git is the only place a fix can actually stick. During this incident, the real fix had to land as a committed, merged change before it survived — the manual patch bought a few minutes at best, and gave a false sense of "it's fixed" that evaporated on the next sync cycle. For an incident under selfHeal, the fastest real path to recovery is a fast-tracked PR, not a live patch.

The Fix, Applied Selectively

Five follow-up PRs, each fixing a specific verified failure mode as it was confirmed live — not a blanket re-revert of everything:

# Reverted only what was proven to break — capabilities stay dropped
# wherever the image's entrypoint doesn't need them:
securityContext:
  allowPrivilegeEscalation: false
  # capabilities.drop: ["ALL"]  ← removed for this specific image,
  # see inline comment for the verified failure mode

# kubernetes/apps/authelia/authelia.yml
securityContext:
  allowPrivilegeEscalation: false
  # SEC-012: image entrypoint runs as root and needs CAP_CHOWN/CAP_SETGID/
  # CAP_SETUID to chown /config and su-exec into its runtime user —
  # confirmed live, dropping all capabilities crash-loops it
  # ("su-exec: setgroups(0): Operation not permitted").

Each reverted file got an inline comment recording the specific verified failure mode — not a vague "this broke things." The next person (or future me) who's tempted to re-attempt a blanket capability drop across this manifest tree has the actual evidence sitting right there, rather than rediscovering it the same way.

Final state: allowPrivilegeEscalation: false everywhere — that one's genuinely always safe, it has no entrypoint-behavior dependency. capabilities.drop: [ALL] kept only where verified safe (cloudflared, gitea after its own fix, and several others). runAsNonRoot: true kept only where the image's actual default user is verifiably non-root. Net result: Trivy's configuration-misconfiguration finding count went from 215 to 171 — real progress, just not the full sweep the first PR claimed.

The Lesson

kubeconform and kubectl --dry-run validate that a manifest is schema-valid. They say nothing about whether the container's actual entrypoint will survive the constraints you just imposed on it. Those are two completely different questions, and passing the first one tells you nothing about the second.

For any image you don't control, the actual behavior of its entrypoint — does it run as root and drop privileges, does it default to a non-root user, does anything in its startup sequence need a specific capability — has to be verified live, one image at a time, before a blanket security hardening change goes anywhere near a cluster with auto-sync enabled. The pattern to specifically watch for: anything that does chown/chmod on a data directory before launching the real process almost certainly needs CAP_CHOWN and friends, regardless of how harmless the final running process looks.

The same blast-radius problem exists in Azure — a blanket Pod Security Standard or Azure Policy applied across an AKS cluster's namespaces can break exactly this class of container for exactly this reason, just with kubectl apply replaced by a policy assignment that enforces on the next pod restart instead of immediately. Verify per-workload before enforcing cluster-wide, not after.

Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS

david — Mon, 22 Jun 2026 12:14:01 +0000

Two Raspberry Pi 4Bs run AdGuard Home and Unbound for an entire home network, in an active/passive pair via Keepalived. They're physical hardware sitting on a shelf, not VMs or LXCs — no Proxmox snapshot, no PBS backup, no terraform destroy && apply to recover from a bad state. If one hangs hard at 2am, nobody notices until someone's phone can't resolve a hostname.

This is the hardening pass that closed every gap I found in that setup: a hardware watchdog for total-system-freeze recovery, fail2ban for the one SSH-exposed surface, an nftables host firewall that's careful not to fight with Docker's own iptables rules, log size caps to stop slow SD-card death, and a DNS health check that works even on the day the rest of the monitoring stack is offline — which, as it turned out, was exactly the day it mattered.

View the complete homelab infrastructure source on GitHub 🐙

Why "It's Just DNS" Needs More Hardening, Not Less

The instinct with a small, single-purpose device is to leave it alone — fewer moving parts, fewer ways to break it. That's backwards for a device with no operator watching it and no automated recovery path. A k3s pod that crashes gets rescheduled in seconds. A Raspberry Pi that hard-hangs stays hung until a human walks over and pulls the power.

Everything below is about closing that gap: detecting failure independently, recovering from total freezes without intervention, and not introducing a new failure mode in the process of doing any of this.

Hardware Watchdog: Recovering From a Hang Software Can't See

A crashed container gets restarted by Docker. A kernel deadlock — the whole system stops responding, nothing crashes, nothing logs anything — doesn't. Nothing is left running to notice the problem or act on it.

The Broadcom SoC in a Raspberry Pi has a hardware watchdog timer: a circuit that resets the board if it isn't periodically "petted." As long as something pets it, the system is presumed alive. If petting stops — because the kernel is deadlocked and nothing can run — the watchdog fires and power-cycles the board.

# /boot/firmware/config.txt
dtparam=watchdog=on

# /etc/systemd/system.conf
RuntimeWatchdogSec=15s
RebootWatchdogSec=10min

RuntimeWatchdogSec=15s means systemd pets the hardware watchdog every 15 seconds while the system is healthy. If systemd itself stops running (the actual deadlock case this exists for), the pets stop, and the watchdog circuit force-resets the board. RebootWatchdogSec=10min is a second, independent safety net — if a reboot itself hangs (stuck somewhere in shutdown), the watchdog fires again after 10 minutes rather than leaving the board hung mid-reboot indefinitely.

This requires a reboot to take effect — the config.txt change only applies at boot. I gated the actual reboot behind an explicit flag (rpi_optimize_reboot, default false) rather than auto-rebooting a DNS server as a side effect of an Ansible run.

fail2ban: The One Exposed Surface

These Pis are reachable from the entire server VLAN, and via the Keepalived VIP, present a single consistent address that's an obvious target for anything scanning the network. The only network-facing attack surface that matters here is SSH.

# /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true
port = ssh
filter = sshd
maxretry = 5
findtime = 10m
bantime = 1h

Five failed attempts within ten minutes bans the source IP for an hour. fail2ban only watches sshd auth logs — it has zero interaction with the DNS path (AdGuard, Unbound, Docker). That isolation matters: a misconfigured fail2ban jail watching the wrong log file, or banning based on the wrong filter, is a self-inflicted outage risk on a box where outages are expensive. Scoping it to exactly one well-understood log source keeps the blast radius of a fail2ban misconfiguration limited to "SSH access," never to DNS itself.

The nftables Trap: Don't Touch /etc/nftables.conf

This is the part that could have caused the exact outage the rest of this hardening pass exists to prevent.

The obvious way to add a host firewall on Debian is to edit /etc/nftables.conf and enable nftables.service. The problem: that file conventionally starts with flush ruleset — and Docker manages its own NAT and FORWARD chains via iptables-nft (the nftables-backed iptables compatibility layer). Enabling the stock nftables.service would flush ruleset on every boot, wiping out Docker's NAT rules along with it, and silently break every published container port. On a box running AdGuard with network_mode: host specifically so it can bind port 53 directly — but also running other containers in bridge mode with published ports — that's not a hypothetical, it's the actual topology.

The fix: don't touch /etc/nftables.conf or the stock service at all. Use a separate ruleset file and a separate, custom systemd service:

# /etc/nftables-hostfw.conf
table inet hostfw {
  chain input {
    type filter hook input priority filter; policy drop;
    iif "lo" accept
    ct state established,related accept
    ip protocol icmp accept
    meta l4proto ipv6-icmp accept
    tcp dport 22 accept
    tcp dport 53 accept
    udp dport 53 accept
    tcp dport 3001 accept
    tcp dport { 80, 443 } accept
    udp dport 41641 accept
    ip protocol vrrp accept
  }
}

# /etc/systemd/system/hostfw.service
[Unit]
Description=Host firewall (inet hostfw table, additive — does not touch Docker's tables)
After=network.target docker.service
Wants=docker.service

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/sbin/nft -f /etc/nftables-hostfw.conf
ExecStop=/usr/sbin/nft delete table inet hostfw

[Install]
WantedBy=multi-user.target

A named table (inet hostfw) in its own namespace, with policy drop only on that table's input chain — it's additive to whatever else nftables is doing, not a replacement of the ruleset. After=docker.service and Wants=docker.service ensure ordering: this table gets applied after Docker has already set up its own rules, so there's no race where this firewall's policy drop briefly applies before Docker's accept rules for its own traffic exist.

What this firewall covers: SSH (22), DNS (53 — AdGuard runs network_mode: host, so this is genuinely host-stack traffic, not Docker-NAT'd), AdGuard's web UI (3001), the HAProxy VIP (80/443), Tailscale (41641/udp), Keepalived VRRP.

What it deliberately doesn't cover: bridge-mode containers like Unbound (5335) and node_exporter (9100). Docker DNATs traffic to these before it ever reaches the host's INPUT chain — this firewall's table never sees that traffic, confirmed by live testing, not just by reading documentation about how Docker's iptables integration works. Restricting bridge-mode container ports would require rules in Docker's own DOCKER-USER chain, with careful IPv4/IPv6 handling to avoid breaking container egress. I deferred this: MikroTik already segments these Pis from the wider internet at the network layer, and the mistake-risk of getting DOCKER-USER chain rules wrong on a live DNS server outweighed the marginal security benefit of restricting traffic that's already internal-only.

Validation that actually validates the deployment path, not just the live change: live-tested on the replica Pi first, with a systemd-run safety-rollback timer staged before every individual change (the same dead-man's-switch pattern as the MikroTik cleanup). Then re-tested via the actual Ansible run — a separate code path from the manual live test, since a playbook can have a templating bug that a manual nft -f test wouldn't catch. Then validated with an actual reboot, to confirm the systemd service correctly reapplies the ruleset on boot, rather than only working because it happened to still be live-applied from the manual test. Only after the replica was fully green did the same sequence run against the primary DNS node.

Stopping Slow SD-Card Death

Docker's default json-file log driver has no size limit. On a box with a real disk, that's eventually a problem; on a Pi with an SD card as its only storage, it's a slow-motion outage that looks like nothing is wrong until the card is full and everything stops:

// /etc/docker/daemon.json
{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Existing container logs were already at 17MB and 2.7MB by the time I checked — not catastrophic yet, but on a trajectory toward "disk full" with zero warning beforehand, months out. This setting only caps logs for containers created or recreated after the daemon restart — it doesn't retroactively truncate what's already there. Existing oversized logs needed a manual one-time cleanup; the daemon-wide default just stops the problem from recurring.

Memory Limits: Catching a Leak Before It Takes the Whole Pi Down

# docker-compose, per service
adguardhome:
  mem_limit: 512m
unbound:
  mem_limit: 256m
promtail:
  mem_limit: 256m
node_exporter:
  mem_limit: 128m
autoheal:
  mem_limit: 64m

These are generous numbers, chosen from actual observed usage with real headroom — the goal isn't to constrain normal operation, it's to make sure a genuine memory leak or runaway process in one container gets killed by Docker's OOM handling for that container before it starves every other process on the Pi, including the DNS resolver everything depends on. Tested incrementally on the replica first, verified via docker inspect that limits were actually enforced, confirmed all containers came back Up after restart, with DNS unaffected throughout — the kind of change where "looks fine" isn't sufficient confirmation on a box this important.

Local Config Backup: The Gap Nobody Noticed

These Pis are physical hardware — Proxmox Backup Server and Velero only cover VMs and LXCs, so neither one was ever backing these up. The gap had existed since the Pis were first deployed, just never surfaced, because nothing had ever required restoring from a backup yet.

#!/bin/bash
# /usr/local/bin/backup-rpi-configs.sh
set -euo pipefail
DEST=/opt/backups
STAMP=$(date +%Y%m%d-%H%M%S)
tar czf "${DEST}/configs-${STAMP}.tar.gz" \
  -C / opt/adguardhome/conf opt/unbound 2>/dev/null || true
ls -t "${DEST}"/configs-*.tar.gz 2>/dev/null | tail -n +15 | xargs -r rm --

Daily, via a systemd timer with randomized delay (to avoid both Pis hitting disk I/O at the exact same instant), keeping the 14 most recent snapshots. Deliberately local-only, with no NFS or git dependency — the NFS server runs as an LXC on the Proxmox host, and depending on the thing you're backing up away from failing defeats the purpose. AdGuard's config also contains a bcrypt password hash; pushing that into git history, even encrypted-at-rest on a private remote, is an unnecessary exposure for a snapshot whose only job is "let me recover the last known-good config after an accidental change."

Alerting That Survives the Main Alerting Stack Being Down

This is the piece that mattered in practice, not just in theory. The homelab's primary alerting path (Prometheus → Alertmanager → Discord) runs on the k3s cluster, which runs on the Proxmox host. On the day I built this, the Proxmox host itself was down for hardware repair — which meant the entire alerting pipeline was also down, on exactly the day DNS health mattered most, since DNS was now also the only thing left running unsupervised.

#!/bin/bash
# Independent DNS health check — ZERO dependency on k3s/Prometheus/Alertmanager
WEBHOOK_URL="..."
STATE_FILE="/var/lib/dns-healthcheck.state"
HOSTNAME=$(hostname)

check_dns() {
  dig +short +timeout=3 google.com @127.0.0.1 -p 53 > /dev/null 2>&1 && \
  dig +short +timeout=3 google.com @127.0.0.1 -p 5335 > /dev/null 2>&1
}

PREV_STATE="unknown"
[ -f "$STATE_FILE" ] && PREV_STATE=$(cat "$STATE_FILE")

if check_dns; then CURRENT_STATE="healthy"; else CURRENT_STATE="unhealthy"; fi

if [ "$CURRENT_STATE" != "$PREV_STATE" ]; then
  if [ "$CURRENT_STATE" = "unhealthy" ]; then
    MESSAGE="🔴 **${HOSTNAME}**: DNS resolution failing. This alert is independent of the main monitoring stack."
  else
    MESSAGE="🟢 **${HOSTNAME}**: DNS resolution recovered."
  fi
  curl -s -X POST -H "Content-Type: application/json" \
    -d "{\"content\": \"${MESSAGE}\"}" "${WEBHOOK_URL}" > /dev/null 2>&1 || true
fi

echo "$CURRENT_STATE" > "$STATE_FILE"

Run every two minutes via a systemd timer. Two design choices that matter more than the script's mechanics:

It tests both layers independently — AdGuard on port 53 and Unbound directly on port 5335. AdGuard forwards to Unbound; testing only the front door (53) wouldn't distinguish "AdGuard is fine but its upstream resolver died" from "everything's fine." && between the two dig calls means both have to succeed for the overall state to be healthy.

It only posts on a state change, not on every run. A naive healthcheck that posts every two minutes regardless of state either spams a channel into being muted (defeating the purpose) or gets its messages ignored after the first few identical ones. Tracking previous state in a file and diffing against it means the alert fires exactly twice per incident: once when it breaks, once when it recovers — and nothing in between.

The webhook URL reuses the same Discord webhook Alertmanager already posts to — found, while wiring this up, to have been committed in plaintext in the cluster's own monitoring config. Worth its own fix, but explicitly out of scope for this change; noted rather than silently expanded into a second unrelated remediation in the same commit.

What Actually Got Tested, Not Just Written

Every change here got the same validation discipline, because the box matters too much to skip it: replica first, primary only after the replica was fully green; a manual live test and a separate Ansible-driven test, since they're different code paths; and for anything that should survive a reboot, an actual reboot — not just trusting that a systemd unit file is correct.

The pattern generalizes past Raspberry Pis: any unattended edge device — a branch-office router, an IoT gateway, a remote sensor node — has the same shape of problem. No operator watching it, no automated platform-level recovery, and a failure mode (hard hang) that ordinary application-level monitoring can't see because the monitoring agent itself is also hung. A hardware watchdog plus an alerting path with zero dependency on the thing being monitored is the minimum bar for "I'll find out if this breaks," regardless of what the device actually does.

IPv6 NAT66 Behind a FritzBox: The RouterOS 7 Bug That Broke WiFi Clients

david — Mon, 22 Jun 2026 12:13:49 +0000

Most homelab IPv6 guides assume you have native IPv6 from your ISP: a delegated /56 prefix, clean RA on the WAN, no NAT. That describes maybe 30% of actual deployments in Germany.

The other 70% sits behind a FritzBox with DS-Lite or CGN, gets a GUA on the WAN interface via SLAAC, and has no delegated prefix to distribute internally. If you want IPv6 inside your network, you build it yourself.

This is the setup I run: ULA addressing internally, NAT66 masquerade for outbound, everything Terraform-managed. It worked until RouterOS 7's router advertisement defaults caused every FritzBox WiFi client to route IPv6 through MikroTik — and then get dropped.

View the complete homelab infrastructure source on GitHub 🐙

The Topology

Internet
    │
FritzBox (CGN / DS-Lite)
    │  ether1 (WAN) — gets GUA via SLAAC from FritzBox
MikroTik RB5009
    ├── vlan10-mgmt   fd10::1/64
    ├── vlan20-srv    fd20::1/64
    ├── vlan30-dmz    fd30::1/64
    ├── vlan40-iot    fd40::1/64
    └── vlan100-admin fd64::1/64

The FritzBox provides:

IPv4 via CGN/DS-Lite (no public IPv4)
IPv6 GUA prefix via RA on its LAN port — MikroTik's ether1 picks this up via SLAAC

Internally, I use ULA (fd00::/8, RFC 4193). ULA is the IPv6 equivalent of RFC1918 private addressing. It's stable — it doesn't change when the ISP rotates the GUA prefix — and it works for all internal communication. The NAT66 rule masquerades ULA sources to the GUA when leaving ether1.

Step 1: ULA Addresses Per VLAN

Each VLAN gets a /64 from the fd::/8 space. I use the VLAN number as the second octet for readability:

# terraform/stacks/network/ipv6_network.tf

locals {
  ipv6_ula_prefixes = {
    "vlan10-mgmt"   = "fd10::/64"
    "vlan20-srv"    = "fd20::/64"
    "vlan30-dmz"    = "fd30::/64"
    "vlan40-iot"    = "fd40::/64"
    "vlan100-admin" = "fd64::/64"
  }
}

resource "routeros_ipv6_address" "vlan_ula" {
  for_each = local.ipv6_ula_prefixes

  address   = replace(each.value, "::/64", "::1/64")
  interface = each.key
  advertise = true
  comment   = "ULA gateway for ${each.key}"
}

advertise = true enables IPv6 ND (Neighbor Discovery) on each interface. Hosts on each VLAN receive a Router Advertisement with the /64 prefix and auto-configure a ULA address via SLAAC. No DHCPv6 needed.

The router address is ::1 in each /64: fd10::1/64, fd20::1/64, etc.

Step 2: Accept RA on ether1

MikroTik defaults to ignoring Router Advertisements when forward = true (i.e., when acting as a router). You have to explicitly enable RA acceptance on the WAN interface:

resource "routeros_ipv6_settings" "global" {
  accept_router_advertisements = "yes"
  forward                      = true
}

With this, ether1 accepts the RA from the FritzBox and configures its GUA via SLAAC. ip6 address print will show the GUA alongside the manually configured ULA if you have any internal IPv6 config on ether1.

Step 3: NAT66 Masquerade

The NAT66 rule masquerades outbound IPv6 from ULA sources to the GUA on ether1:

resource "routeros_ipv6_firewall_nat" "nat66_masquerade" {
  chain         = "srcnat"
  action        = "masquerade"
  src_address   = "fd00::/8"
  out_interface = "ether1"
  comment       = "NAT66: ULA → WAN GUA (FritzBox upstream)"
}

The src_address = "fd00::/8" constraint is critical. Without it, the rule matches ALL IPv6 traffic leaving ether1 — including traffic from FritzBox WiFi clients that happens to transit MikroTik. This is one half of the bug that caused problems (more on that below).

Step 4: IPv6 Firewall

The IPv6 firewall mirrors the IPv4 firewall philosophy: default-drop, explicit allows, place_before for deterministic rule ordering.

# INPUT chain
resource "routeros_ipv6_firewall_filter" "v6_in_00_established" {
  action           = "accept"
  chain            = "input"
  connection_state = "established,related,untracked"
  place_before     = routeros_ipv6_firewall_filter.v6_in_01_icmpv6.id
  comment          = "V6-IN-00: Allow established/related"
}

resource "routeros_ipv6_firewall_filter" "v6_in_01_icmpv6" {
  action       = "accept"
  chain        = "input"
  protocol     = "icmpv6"
  place_before = routeros_ipv6_firewall_filter.v6_input_drop_all.id
  comment      = "V6-IN-01: Allow ICMPv6 (NDP, RA, ping6)"
}

resource "routeros_ipv6_firewall_filter" "v6_input_drop_all" {
  action  = "drop"
  chain   = "input"
  comment = "V6-IN-DROP: Drop all other IPv6 input"
}

# FORWARD chain
resource "routeros_ipv6_firewall_filter" "v6_fwd_00_established" {
  action           = "accept"
  chain            = "forward"
  connection_state = "established,related,untracked"
  place_before     = routeros_ipv6_firewall_filter.v6_fwd_01_icmpv6.id
  comment          = "V6-FWD-00: Allow established/related"
}

resource "routeros_ipv6_firewall_filter" "v6_fwd_01_icmpv6" {
  action       = "accept"
  chain        = "forward"
  protocol     = "icmpv6"
  place_before = routeros_ipv6_firewall_filter.v6_fwd_02_internal_out.id
  comment      = "V6-FWD-01: Allow ICMPv6"
}

resource "routeros_ipv6_firewall_filter" "v6_fwd_02_internal_out" {
  action        = "accept"
  chain         = "forward"
  src_address   = "fd00::/8"
  out_interface = "ether1"
  place_before  = routeros_ipv6_firewall_filter.v6_forward_drop_all.id
  comment       = "V6-FWD-02: Allow internal ULA to WAN"
}

resource "routeros_ipv6_firewall_filter" "v6_forward_drop_all" {
  action  = "drop"
  chain   = "forward"
  comment = "V6-FWD-DROP: Drop all other IPv6 forward"
}

The forward rule v6_fwd_02_internal_out only allows ULA sources (fd00::/8) to exit via ether1. That's intentional — and it's what exposed the RouterOS 7 bug.

The Bug: RouterOS 7 Sends RA on All Interfaces

After deploying this configuration, FritzBox WiFi clients started losing IPv6 connectivity.

The symptom: devices on the FritzBox WiFi (SSID, not the MikroTik VLANs) had IPv6 addresses but couldn't reach the internet via IPv6. traceroute6 on an affected device showed the path going through MikroTik — not the FritzBox.

The cause: RouterOS 7 enables Router Advertisement on all interfaces by default, including ether1 (WAN).

Here's the sequence:

MikroTik receives a GUA prefix from FritzBox via RA on ether1
RouterOS 7 then re-advertises a Router Advertisement on ether1 — back towards the FritzBox
The FritzBox sees MikroTik advertising itself as an IPv6 router on the LAN
FritzBox WiFi clients pick up MikroTik's RA and install it as their default IPv6 gateway
IPv6 traffic from WiFi clients routes through MikroTik's FORWARD chain
FORWARD chain only accepts fd00::/8 sources — GUA addresses from WiFi clients don't match
Traffic dropped. IPv6 broken for all FritzBox WiFi clients.

The fix is to disable RA on ether1. In RouterOS /ip6/nd, find the ether1 entry and set advertise=no.

The problem: as of terraform-routeros provider version 1.99.1 (latest at time of writing), there is no routeros_ipv6_nd resource to manage this via Terraform. The fix has to be applied manually:

/ipv6/nd set [find interface=ether1] advertise=no

This is documented in the Terraform configuration as a comment so it doesn't get overwritten by a future terraform apply:

# RouterOS 7 enables RA advertisement on ALL interfaces by default — including
# ether1 (WAN). Once ether1 gets a GUA via SLAAC, MikroTik starts sending RAs
# on the FritzBox LAN. FritzBox WiFi clients then use MikroTik as their IPv6
# gateway, but the FORWARD chain only allows fd00::/8 sources → GUA clients
# are dropped → IPv6 broken on FritzBox WiFi.
# RA on ether1 is disabled in RouterOS: /ipv6/nd set [find interface=ether1] advertise=no
# routeros_ipv6_nd is not exposed in terraform-routeros/routeros ≤ 1.99.1 (latest as of 2026-06).

Once routeros_ipv6_nd is added to the provider (tracked upstream), this should be managed as:

resource "routeros_ipv6_nd" "ether1_no_ra" {
  interface = "ether1"
  advertise = false
}

ULA vs. GUA: Why Not Just Use the ISP Prefix?

The obvious alternative: use the GUA prefix the FritzBox receives from the ISP, delegate a /64 to each VLAN, and skip NAT66 entirely. IPv6 was designed to eliminate NAT.

The problem: German ISPs frequently rotate GUA prefixes. A prefix change means every device on every VLAN gets a new address — breaking DNS records, Ansible inventory, firewall rules, and anything else that references addresses directly.

ULA solves this. The fd::/8 prefix is locally assigned and never changes. Internal addressing is stable forever. The NAT66 rule handles the GUA ↔ ULA translation at the WAN boundary transparently.

The trade-off: ULA + NAT66 breaks end-to-end IPv6 reachability (GUA hosts on the internet can't initiate connections to your ULA hosts). For a homelab where all inbound connections come through a Cloudflare Tunnel or Traefik ingress anyway, that's not a problem.

Verifying the Setup

After applying the Terraform config and the manual RA fix:

# From a device on vlan20-srv (should have fd20::/64 address)
ip -6 addr show
# Should see: fd20::xxx/64

# Test outbound IPv6
ping6 -c 3 ipv6.google.com
# Should succeed (NAT66 masquerades the ULA source to the GUA)

# From a FritzBox WiFi device
ip -6 route show
# Default via should point to FritzBox, not MikroTik

If WiFi clients still route through MikroTik after setting advertise=no, run ip6/nd print on the RouterOS terminal to verify the change persisted. RouterOS can be slow to propagate ND configuration changes.

The same ULA-vs-GUA stability trade-off shows up in Azure networking — except there it's RFC1918 address space behind NAT Gateway or Azure Firewall instead of a CGN ISP. If you're designing the equivalent zero-trust network layer for Azure, the same default-deny-plus-explicit-allow philosophy applies.

My Firewall Had 77 Rules. Terraform Knew About 22 of Them.

david — Sun, 21 Jun 2026 15:43:57 +0000

I wrote an article about building a zero-trust MikroTik firewall with Terraform — default-deny chains, explicit allow rules, place_before for deterministic ordering. The Terraform code was correct. I'd run terraform plan regularly and it showed no drift.

The live router had 77 firewall filter rules. The Terraform configuration tracked 22.

This is the story of how that happened, why terraform plan showing clean didn't catch it, and how a security tightening I'd made — and verified, and considered done — had been silently undone for weeks.

View the complete homelab infrastructure source on GitHub 🐙

How You End Up With Four Generations of the Same Firewall

The pattern, in hindsight, is obvious: every time I did a significant firewall rework, I wrote a fresh, complete set of rules in firewall_deterministic.tf and ran terraform apply. Terraform created the new rules. It did not — because nothing told it to — remove the old generation, because the old generation's rules weren't Terraform resources Terraform knew about. They'd been created by a previous terraform apply of an earlier version of the same file, then the resource definitions were edited or replaced rather than removed cleanly, or in a couple of cases, created directly via the RouterOS API during a debugging session and never imported.

Terraform only manages what's in its state. A rule that exists on the router but isn't a resource in the current configuration is invisible to terraform plan — there's no diff to show, because there's nothing in the config to compare it against. terraform plan reporting "no changes" means the resources Terraform knows about match reality. It says nothing about resources Terraform was never told to track.

The Bug This Actually Caused

This wasn't just clutter. It actively undid a real security fix.

At some point I'd tightened a monitoring rule from "Prometheus can reach all internal VLANs" to "Prometheus can reach only port 9100 on the management VLAN":

# The narrow, intentional version — added to fix an overly broad rule
resource "routeros_ip_firewall_filter" "fwd_04a_srv_monitoring" {
  action       = "accept"
  chain        = "forward"
  src_address  = "10.0.20.0/24"
  dst_address  = "10.0.10.0/24"
  dst_port     = "9100"
  protocol     = "tcp"
  place_before = routeros_ip_firewall_filter.fwd_08_allow_dns.id
  comment      = "04a: SRV - Prometheus scrape to MGMT node_exporter (port 9100)"
}

This rule existed in Terraform. terraform plan showed it as applied, no drift. I had every reason to believe the network was scoped exactly this way.

But RouterOS evaluates firewall rules in order and stops at the first match. Buried earlier in the live ruleset — a leftover from a previous generation — was the old, broad version:

"04a: SRV - Allow monitoring to all internal VLANs"
src=10.0.20.0/24 dst=10.0.10.0/24 action=accept

No port restriction. No protocol restriction. And because RouterOS hit this rule first, traffic matching it was accepted before the router ever evaluated the narrower, newer rule. The port-9100-only restriction I'd written, tested, and confirmed in Terraform had never actually been enforced on the live device — the older, broader rule was silently winning every time.

This is the sharpest version of the general problem with ordered rule lists: a rule that looks dead (superseded by a newer one) isn't dead unless it's actually removed. It's just sitting there, waiting for the day its broader match happens to fire first.

Finding the Actual Scope of the Problem

# Pull live rules via the RouterOS REST API
curl -s -k -u admin:$PASS https://10.0.10.1/rest/ip/firewall/filter | jq length
# → 77

# Count Terraform-managed resources
grep -c 'resource "routeros_ip_firewall_filter"' terraform/stacks/network/firewall_deterministic.tf
# → 22

55 rules existed on the router with no corresponding Terraform resource. Diffing live rules against the 22 known-good ones by exact field match (action, chain, src/dst address, port, protocol — not just comment text, since comments had also drifted across generations) split that 55 into two groups:

36 rules were exact or near-exact duplicates of a currently-tracked rule — leftover generations of the same intent, just stale.
19 rules were legitimate, distinct, and still in active use — VPN access tiers, Atlantis/MikroDash API access, WireGuard, a Minecraft server port-forward, OIDC redirect routes. These had been created manually at some point and simply never added to Terraform in the first place. Not drift in the dangerous sense — just infrastructure that was never brought under IaC.

Deleting Firewall Rules Without Locking Yourself Out

This is the highest-blast-radius device in the network. A mistake deleting the wrong rule doesn't get fixed by SSHing back in — if the rule that breaks is the one allowing SSH, there's no way back in remotely. Before deleting anything, I staged a full per-rule restore as a one-shot RouterOS scheduler entry — a dead man's switch:

/system scheduler add name="restore-firewall-failsafe" \
  start-time=startup interval=5m \
  on-event="/system script run restore-firewall-rules"

The restore script re-creates every rule about to be deleted, scheduled to fire automatically in five minutes unless cancelled. The procedure:

Stage the restore script and the scheduler entry (not yet running — disabled=yes).
Enable the scheduler.
Delete the 36 orphaned rules via direct REST API calls.
Immediately verify DNS, SSH, and WAN connectivity from a separate, already-open session.
Only if everything checks out: disable and remove the scheduler entry.

If step 4 had failed — if deleting a rule had broken something — the scheduler would have restored the deleted rules automatically within five minutes, without requiring any further access to the router. This pattern generalizes to any change where the failure mode is "I can no longer reach the device to fix my mistake": stage the rollback to fire automatically on a timer, and only cancel the timer after confirming success through a separate channel.

What's Left

41 rules remain: the 22 Terraform-managed ones, plus the 19 legitimate manual rules — now tracked as a known gap (docs/OPERATIONS.md) rather than invisible clutter. Bringing those 19 under Terraform via import blocks is the obvious next step, but it's explicitly not urgent — they're working, intentional, and visible in documentation now. The 36 that mattered (because they were actively undermining a security control) are gone.

The General Lesson

terraform plan showing no drift is not the same claim as "the live device matches my intent." It only means the resources Terraform is tracking match their last-applied state. Anything created outside that tracked set — via a prior version of the config that got edited rather than cleanly replaced, or via direct API/CLI access during a debugging session — is invisible to the diff, indefinitely, until someone goes and looks at the live device directly.

For an ordered rule list specifically (firewalls, but also things like Azure Firewall Policy rule collections, NSG priority-ordered rules, or any first-match system), an orphaned broad rule isn't neutral clutter — it can silently take precedence over a narrower rule you believe supersedes it. Periodically diffing live state against Terraform state by direct query — not just trusting plan — is the only way to catch this class of bug.

The same risk exists in Azure NSGs and Azure Firewall Policy: priority-ordered rules where an old, broad rule with a lower priority number can silently win over a newer, narrower one if it was never cleaned up after a security tightening. If you're managing NSG rule sets at scale, periodically pulling live rule state via az network nsg rule list and diffing it against your Terraform state catches exactly this class of drift before it becomes a finding in someone else's audit.

Kyverno: Supply Chain Security as Admission Control on Kubernetes

david — Sun, 21 Jun 2026 15:43:51 +0000

Kubernetes has no opinion about what you run. You can deploy a container with no resource limits, no security context, root access to the host filesystem, and an image tagged :latest that changes every week — and the scheduler will place it without complaint.

In a homelab that's annoying. In a production cluster, it's a compliance failure.

Kyverno is a Kubernetes-native policy engine. It runs as an admission webhook — every kubectl apply, every ArgoCD sync, every Helm install is evaluated against your policies before it reaches the scheduler. Violations are either blocked (Enforce) or logged (Audit).

This post covers the three policies running on my k3s cluster and the Audit-first rollout strategy that lets you enforce gradually without breaking existing workloads.

View the complete homelab infrastructure source on GitHub 🐙

Why Admission Control

The alternative to admission control is runtime enforcement: scan running containers, alert on violations, remediate manually. This works, but it's reactive. A misconfigured deployment reaches the scheduler, requests a node, pulls an image, and starts running before anything flags it.

Admission control is preventive. The webhook intercepts the API request before the object is created. A rejected request never touches the scheduler.

For supply chain security specifically — controlling what can run, not just what is running — admission control is the right layer.

Installing Kyverno via ArgoCD

Kyverno deploys via Helm:

# kubernetes/system/kyverno/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://kyverno.github.io/kyverno/
    targetRevision: 3.2.6
    chart: kyverno
    helm:
      values: |
        admissionController:
          replicas: 1
        backgroundController:
          replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

The backgroundController is the component that evaluates existing resources against policies (background scan) and populates PolicyReport objects. Without it, you only catch violations at admission time — existing non-compliant resources stay invisible.

Policy 1: Require Resource Limits (Audit)

Containers without resource limits are a noisy-neighbour problem. A single container that consumes unbounded memory will trigger the OOM killer across the whole node, affecting every other pod on it.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
  annotations:
    policies.kyverno.io/title: Require Resource Limits
    policies.kyverno.io/description: >
      Pods without resource limits can starve other workloads on the same node.
      Run `kubectl get policyreport -A` to see violations before enforcing.
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: check-container-limits
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [apps, monitoring, database]
      validate:
        message: "Container '{{ request.object.spec.containers[0].name }}' must define resources.limits (cpu and memory)."
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.resources.limits | length(@) }}"
                    operator: Equals
                    value: 0

This is in Audit mode — violations are logged to PolicyReport but not blocked. That's intentional. Before enforcing, you need to know what would break.

Check current violations:

kubectl get policyreport -A
kubectl describe policyreport -n apps

The report shows every pod in apps, monitoring, and database namespaces that's missing resource limits. Fix those, then flip validationFailureAction to Enforce.

Policy 2: Disallow Privileged Containers (Enforce)

This one is in Enforce mode from day one. No homelab service needs privileged mode — if something requires privileged: true, that's a flag to investigate, not accommodate.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged-containers
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-privileged
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [apps, monitoring, database, external-secrets]
      validate:
        message: "Privileged containers are not allowed."
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.securityContext.privileged || false }}"
                    operator: Equals
                    value: true

|| false handles the case where securityContext is not set — the expression evaluates to false, which correctly passes validation (no security context means non-privileged).

This blocks:

Containers with securityContext.privileged: true
By extension, anything that tries to use hostPID, hostNetwork, or hostPath in ways that require privilege escalation

Policy 3: Disallow `:latest` Image Tag (Audit)

Images tagged :latest are non-deterministic. What nginx:latest points to today is different from what it pointed to last week. Rollbacks are impossible because you can't pin to the previous image. Reproducible deployments require pinned tags — either semver (4.39.20) or digest (sha256:abc123).

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Audit
  background: true
  rules:
    - name: check-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [apps, monitoring, database]
      validate:
        message: "Image '{{ element.image }}' must use a specific tag, not :latest or untagged."
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.image }}"
                    operator: Equals
                    value: "*:latest"
                  - key: "{{ element.image | contains(@, ':') }}"
                    operator: Equals
                    value: false

The second condition catches images with no tag at all — nginx without :latest still resolves to latest under the hood.

Real-World Example: The Authelia `:latest` Violation

When I first enabled the disallow-latest-tag policy in Audit mode and ran kubectl get policyreport -n apps, Authelia showed up immediately:

PASS  disallow-privileged-containers  authelia-xxx    apps
FAIL  disallow-latest-tag             authelia-xxx    apps
  → ghcr.io/authelia/authelia:latest

The Authelia deployment had been running latest from day one. The fix was straightforward:

# Before
image: ghcr.io/authelia/authelia:latest

# After
image: ghcr.io/authelia/authelia:4.39.20

Once pinned, the PolicyReport cleared. This is the Audit → Enforce workflow in practice: enable, observe, fix violations, then enforce.

The Audit → Enforce Rollout Strategy

The pattern that makes Kyverno safe to adopt on a running cluster:

1. Deploy all policies in validationFailureAction: Audit
2. Wait for background scans to populate PolicyReports (5-10 min)
3. kubectl get policyreport -A → review violations
4. Fix violations in affected deployments
5. Change policy to validationFailureAction: Enforce
6. Verify no new violations in PolicyReport

Going directly to Enforce on a running cluster breaks things. ArgoCD sync jobs, Helm hooks, system daemonsets — they all create pods and will hit the policy. Audit first gives you visibility without the blast radius.

Scoping Policies to Specific Namespaces

Notice all three policies match only [apps, monitoring, database]. System namespaces (kube-system, kube-public, argocd, kyverno) are excluded deliberately.

System components often need exception behaviour — hostPath volumes for kubelet, privileged containers for CNI plugins, no resource limits on critical infrastructure. Scoping your policies to user workload namespaces avoids blocking cluster internals while still enforcing what matters.

PolicyReport: The Visibility Layer

# All reports across all namespaces
kubectl get policyreport -A

# Detail for a specific namespace
kubectl describe policyreport polr-ns-apps -n apps

# All violations
kubectl get policyreport -A -o json | \
  jq '.items[].results[] | select(.result == "fail")'

PolicyReports are Kubernetes-native objects. You can build Grafana dashboards against them (Kyverno exports metrics to Prometheus), alert on policy violations, and track compliance trends over time.

The require-resource-limits violation count is a useful metric: as you fix deployments, it should trend towards zero. When it hits zero and stays there, flip to Enforce.

Enterprise Bridge

These three policies map directly to supply chain security requirements under ISO 27001 (A.12.6 — Technical Vulnerability Management) and NIS2 Article 21 (4.b — handling of incidents, 4.e — supply chain security).

In Azure environments, the equivalent layer is Azure Policy + Defender for Containers — the same concept, different implementation. For teams deploying Kubernetes workloads in regulated environments, Kyverno policies committed to Git provide the audit trail that compliance frameworks require: every policy change is a pull request, every violation is logged.

I Ran Gitleaks Against My Own Repo and Found 12 Real Secrets

david — Sun, 21 Jun 2026 15:14:09 +0000

I assumed my homelab repo was clean. No one had ever flagged anything in review (there is no one else reviewing it), CI was green, and I generally try to use Vault and ExternalSecrets for anything sensitive.

Then I ran a full-history gitleaks detect against it. It found 12 distinct secrets committed in plaintext — including the OIDC private key that signs SSO tokens for half the cluster.

This is the scanning setup I put in place afterward, the baseline strategy that let me adopt secret scanning without getting blocked by my own history on every commit, and the remediation plan for the leaks themselves.

View the complete homelab infrastructure source on GitHub 🐙

What Gitleaks Found

gitleaks detect --no-banner -v

Twelve real findings, plus one already-hashed password (lower severity but still shouldn't be hand-committed) and one false positive in ROADMAP.md (documentation text that happened to match a generic API key pattern).

The real findings, by severity:

File	Secret	Why It Matters
`kubernetes/apps/authelia/configmap.yml`	OIDC issuer private key	Signs SSO tokens for ArgoCD, Vault, Grafana — highest blast radius
`kubernetes/apps/garage/config.yml`	RPC secret + admin token	Storage backend for Velero/Loki/CNPG backups
`kubernetes/apps/garage/secrets.yml`	Admin token (duplicate)	Same secret committed twice in two files
`terraform/stacks/network/local_backend.hcl`	Garage S3 access key	This is the Terraform state backend's own credential
`kubernetes/system/postgres/cnpg-backup-secret.yml`	Garage S3 secret key	Used for WAL archiving
`kubernetes/apps/paperless/secrets.yml`	Postgres password + AI API token
`kubernetes/apps/cloudflared/secrets.yml`	Cloudflare Tunnel token
`kubernetes/apps/headscale/config.yml`	OIDC client secret	Must match Authelia's client config
`kubernetes/system/monitoring/loki.yml`	Minio/S3 password
`kubernetes/apps/mikrodash/secrets.yml`	Dashboard password	Lowest priority — internal tool only

None of these were exposed by a public repo (this one is private), but "private repo" is not a security control — it's a single permission setting away from being public, and anyone with read access to the repo (or its history, forever) has all of this regardless.

Why a Private Repo Doesn't Make This Fine

The honest reason these accumulated: early in the project, before Vault and ExternalSecrets were set up, every new service got a quick secrets.yml with the actual values inline, "just to get it working." Once Vault was running, new services went through it — but nobody went back and migrated the old ones. Each individually felt low-risk at the time. Twelve of them, four months later, is a real exposure if the repo's access list ever changes.

This is the same drift pattern as the Terraform-vs-RouterOS-firewall divergence I wrote about separately: each shortcut is locally reasonable, the accumulated state is not.

Setting Up Gitleaks Without Getting Blocked by History

The naive approach — turn on gitleaks protect in pre-commit and call it done — fails immediately. Every single future commit gets blocked by the 12 pre-existing leaks, because gitleaks scans the whole working tree, not just your diff. You'd have to fix all 12 before you could make any other commit, including the commit that adds the scanning.

The fix is a baseline file:

gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v

A baseline is a snapshot of currently-known findings. Anything in the baseline is allowed to keep existing; anything new fails the hook. Generate it once:

gitleaks detect --report-format json --report-path .gitleaks-baseline.json

Commit that baseline file. From this point forward, gitleaks only blocks genuinely new secrets — exactly what you want when adopting scanning on a repo with history older than the scanning itself.

The Three-Layer Hook Setup

One scan layer is not enough — a single missed git commit --no-verify or a commit made from a machine without the hooks installed slips through. Three layers, increasing scope, decreasing frequency:

# .pre-commit-config.yaml
- id: gitleaks-staged
  name: Gitleaks (staged changes)
  description: >-
    Blocks committing NEW secrets. Uses .gitleaks-baseline.json so the
    12 pre-existing leaks don't block every commit until they're fully
    remediated — only genuinely new secrets fail this.
  entry: bash -c 'gitleaks protect --staged --baseline-path .gitleaks-baseline.json --no-banner -v'
  language: system
  always_run: true
  pass_filenames: false
  stages: [pre-commit]

- id: gitleaks-full-repo
  name: Gitleaks (full history, pre-push only)
  description: Re-scans the entire repo and history before any push, against the same baseline.
  entry: bash -c 'gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v'
  language: system
  always_run: true
  pass_filenames: false
  stages: [pre-push]

gitleaks protect --staged at commit time — fast, scans only what's staged, catches a secret before it ever enters history.

gitleaks detect at push time — re-scans the entire repo (slower, but only runs once per push, not once per commit). This catches anything that slipped past the first layer, for example a commit made with git commit --no-verify.

CI runs the same gitleaks detect command as a third, environment-independent layer — catches anything pushed from a machine that never had the hooks installed at all.

Allowlisting Real False Positives

The ROADMAP.md false positive needed an explicit allowlist entry, not a baseline bypass — baseline entries are meant for things you intend to fix, allowlist entries are for things that were never secrets in the first place:

# .gitleaks.toml
[extend]
useDefault = true

[allowlist]
description = "Known false positives"
regexes = [
  # ROADMAP.md doc text listing which services use which OIDC client auth
  # method — matches the generic-api-key pattern but is plain documentation,
  # not a secret.
  '''Proxmox/PBS/Grafana/Headscale use `client_secret_basic`''',
]

Be specific with allowlist regexes. A broad pattern here defeats the entire point of scanning — match the exact false-positive string, not a category of strings that happens to include it.

The Remediation Plan

Finding the leaks and remediating them are two different projects. Remediation means: rotate the actual credential (not just remove it from the file — the old value is still valid until rotated), and move the new value into Vault behind an ExternalSecret so it never gets hand-committed again.

The tricky part is ordering. Some of these credentials are dependencies of each other:

1. Garage RPC secret + admin token + S3 keys
   ↳ Everything else's backups depend on Garage being internally consistent.
     Rotating the S3 key also invalidates Terraform's own state backend
     credential (terraform/stacks/network/local_backend.hcl uses the same
     key) — update both in the same pass or Terraform loses access to its
     own state.

2. Authelia OIDC issuer private key
   ↳ Highest blast radius if left exposed (signs every SSO session).
     After rotating, every service trusting the old key should be checked
     for unexpected active sessions.

3. Everything else, any order
   ↳ Cloudflare Tunnel token (rotate in Cloudflare dashboard first, update
     second — order matters for tokens with an external source of truth).
   ↳ Headscale OIDC client secret must be rotated in lockstep with
     Authelia's matching client config — they're a pair.

A secret with downstream dependents must be rotated with the dependents in mind, not in isolation. Rotating Garage's S3 key without immediately updating the Terraform backend config doesn't remove a vulnerability — it breaks Terraform's access to its own state.

Confirming Remediation Actually Worked

After moving a secret to Vault and rotating the credential, re-run the same scan:

gitleaks detect --baseline-path .gitleaks-baseline.json --no-banner -v

The secret will still show up — it's in history, and the baseline still lists it. That's expected; the baseline isn't meant to disappear until every listed item has actually been fixed. Only regenerate the baseline once all 12 are addressed, as a final confirmation step that nothing was missed in the process — not as a way to make individual items "go away" faster.

What This Doesn't Fix

Scanning catches secrets in files. It does not:

Scrub git history. The old values remain readable to anyone with repo access, forever, unless you rewrite history (git filter-repo) — which has its own risks if anyone else has a clone.
Replace rotation. A secret found and removed from the current file tree is still valid until you change the actual credential at its source (Cloudflare dashboard, Garage admin CLI, Postgres ALTER USER, etc.).
Catch secrets gitleaks' default ruleset doesn't recognize. Custom internal token formats need custom regex rules — useDefault = true covers known formats (AWS keys, generic API key patterns, JWTs) but not everything.

The same baseline-adoption pattern applies directly to any enterprise repo with years of history and no prior secret scanning — which describes most codebases that predate a security initiative. The Vault + ExternalSecrets target architecture this remediation moves toward is the same pattern covered in External Secrets Operator + HashiCorp Vault — that's where these 12 secrets are headed.

ArgoCD Gotchas: Cache Staleness and the SharedResourceWarning Nobody Explains

david — Sun, 21 Jun 2026 15:14:06 +0000

kubectl apply reports success. You check the resource — the field you just changed is back to its old value. No error. No event. kubectl get shows the change applied, then a few seconds later shows it gone, like it never happened.

This isn't a typo or a YAML indentation bug. It's ArgoCD's selfHeal doing exactly what it's designed to do — re-applying from its own cached understanding of what the resource should be, which can lag behind a change you just made by hand, or even behind a fresh git push.

This hit the same homelab three times in one day, across three unrelated resources. Here's the pattern, the fix, and a second, related gotcha that produces a different symptom from a similar root cause.

View the complete homelab infrastructure source on GitHub 🐙

The Symptom

Three separate incidents, same shape:

A Tempo PersistentVolumeClaim's storageClassName kept reverting after being changed.
Traefik's tlsStore and dashboard configuration reverted after a Helm values update.
A paperless-gpt deployment's volumeMounts reverted after a direct edit.

Each time, the sequence was: edit the live resource or push a change to Git → confirm the change is live → come back later → the old value is back, with no error logged anywhere obvious.

Why This Happens: `selfHeal` Plus a Stale Cache

ArgoCD's selfHeal: true continuously reconciles the live cluster state against ArgoCD's rendered understanding of what the Application's manifests/Helm chart should produce. That's the entire point of GitOps — drift gets corrected automatically, so a manual kubectl edit doesn't silently become the new permanent state.

The bug isn't that selfHeal exists. It's that the rendered understanding ArgoCD reconciles against comes from the argocd-repo-server's manifest/Helm chart cache, and that cache doesn't always get invalidated promptly after a fresh git push or a fresh kubectl apply made outside ArgoCD. For a window of time — usually short, but long enough to be confusing — ArgoCD's source of truth for "what should this look like" is stale, and selfHeal faithfully reverts your change back to match it.

This is functionally indistinguishable, from the outside, from "ArgoCD is ignoring my change" — but the actual mechanism is "ArgoCD is enforcing an outdated cached version of what it thinks I want."

The Fix: Force a Hard Refresh

kubectl patch application <name> -n argocd --type merge \
  -p '{"metadata":{"annotations":{"argocd.argoproj.io/refresh":"hard"}}}'

The hard refresh value (as opposed to normal) tells ArgoCD to bypass the repo-server's manifest cache entirely and re-render from source. Wait roughly 15 seconds, then re-check.

If that alone doesn't resolve it, the cache itself may need restarting, not just invalidating for one Application:

kubectl rollout restart deployment argocd-repo-server -n argocd

This is a bigger hammer — it affects every Application's next reconciliation, not just the one you're debugging — so try the targeted hard refresh annotation first.

The StatefulSet Exception

For the Tempo PVC specifically, neither of the above fully resolved it on the first try, because volumeClaimTemplates on a StatefulSet are immutable — Kubernetes rejects any attempt to change them on an existing object. Clearing ArgoCD's stale cache fixes ArgoCD's intent going forward, but it can't retroactively fix a field that was never mutable on the live object in the first place.

The fix there is to delete and recreate the StatefulSet itself (the underlying PVC and its data survive deleting the StatefulSet, as long as you don't also delete the PVC):

kubectl delete statefulset <name> -n <namespace> --cascade=orphan
# re-sync from ArgoCD to recreate the StatefulSet with the new template

--cascade=orphan deletes the StatefulSet object without deleting the Pods or PVCs it owns — letting ArgoCD's next sync recreate the StatefulSet (now with the corrected, non-stale template) and re-adopt the existing PVC.

A Second, Different-Looking Bug With a Related Cause: SharedResourceWarning

A related but distinct symptom: a resource flickers between two different specs, or gets pruned entirely, and .status.conditions on one of the Applications shows a SharedResourceWarning.

This isn't a cache problem — it's an ownership conflict. Two different ArgoCD Applications are both trying to manage a resource with the same name and namespace. In this case: a Helm chart's own ingressRoute.dashboard.enabled flag was creating a Traefik dashboard IngressRoute, while a separately, manually-defined IngressRoute with the same name existed in a different Application's manifest set — both claiming ownership of the same object.

ArgoCD has no way to know which one is "correct" — it just observes that the live object doesn't match what either Application individually expects, and flags the conflict rather than guessing.

The fix is to pick exactly one owner and have the other stop claiming the resource:

# kubernetes/system/traefik/application.yml — Helm chart's own dashboard route, disabled
helm:
  values: |
    ingressRoute:
      dashboard:
        enabled: false  # the manual, Authelia-protected route below is canonical

# kubernetes/system/other-ingressroute.yml — the manually-defined route, kept
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard
  namespace: traefik
spec:
  # ... Authelia-protected route — this is the one that stays

Once only one Application's manifest set defines the object, recreate it (delete the now-orphaned duplicate definition's effect, let the remaining owner's next sync take over cleanly) and the warning clears.

Telling the Two Apart

Symptom	Likely Cause	Fix
A field reverts within seconds of a manual or git-pushed change; no error anywhere	Repo-server cache staleness	`hard` refresh annotation; restart `argocd-repo-server` if that's not enough
A field reverts but `volumeClaimTemplates` is involved on a StatefulSet	Cache staleness plus an immutable field that can't be patched in place	Same cache fix, plus delete-and-recreate the StatefulSet with `--cascade=orphan`
A resource flickers between two different specs, or gets pruned; `SharedResourceWarning` in `.status.conditions`	Two Applications both claim ownership of the same resource	Disable one owner's claim (Helm flag or manifest removal), keep the other

The diagnostic tell: cache staleness is temporal — the same Application reverts a change made moments ago, and a refresh fixes it. Ownership conflict is structural — check .status.conditions for SharedResourceWarning first; if it's there, refreshing the cache won't help, because there's nothing stale about either Application's understanding — they're both correctly rendering their own manifests, and the manifests themselves conflict.

The cache-staleness pattern is specific to ArgoCD's repo-server architecture, but the ownership-conflict pattern is universal to any GitOps tool managing Kubernetes resources — Flux has the same failure mode if two Kustomizations or HelmReleases both define a resource with the same identity. Checking .status.conditions before assuming a sync or cache problem saves a lot of time chasing the wrong fix.

How a 1 GiB Memory Limit Took Down My Entire k3s Cluster

david — Thu, 18 Jun 2026 19:08:07 +0000

It started with Paperless-ngx crashing.

It ended with my control-plane node sitting at a load average of 90, CoreDNS generating 1.2 million DNS queries per day, and worker nodes reporting 3.8 GiB of allocatable memory instead of the 16 GiB they actually had.

The root cause of all of it: a single 1 GiB memory limit set three months earlier without much thought.

This is the full post-mortem — not the sanitized version where everything was obvious in hindsight, but the actual sequence of failures and how I traced each one back to its cause.

View the complete homelab infrastructure source on GitHub 🐙

The Setup

Three-node k3s cluster running on Proxmox VMs (VLAN 20, server subnet):

vm-srv-k3s-11 — control-plane, 4 cores, 12 GiB dedicated
vm-srv-k3s-12 — worker, 4 cores, up to 16 GiB (balloon)
vm-srv-k3s-13 — worker, 4 cores, up to 16 GiB (balloon)

Apps namespace runs about 20 workloads: Nextcloud, Authelia, Paperless-ngx, Jellyfin, Home Assistant, Gitea, Mealie, and more. GitOps via ArgoCD; Longhorn for distributed storage.

Failure 1: Paperless OOMKilled 16 Times in 5 Hours

Paperless-ngx uses Tesseract for OCR and Apache Tika for document ingestion. When a batch of documents hits at once — invoice exports, scanned PDFs — both workers burst memory hard and fast.

The deployment had this:

resources:
  requests:
    memory: 512Mi
    cpu: 250m
  limits:
    memory: 1Gi
    cpu: 500m

That 1 GiB ceiling is too low. When Tesseract processes a high-resolution scanned document, it easily needs 2–3 GiB. The kernel OOM killer terminates the container every time. Kubernetes restarts it. The next document in the queue triggers another OOM. Repeat sixteen times.

Fix: raised limits and reduced concurrency to stay under the higher ceiling:

resources:
  requests:
    memory: 1Gi
    cpu: 500m
  limits:
    memory: 3Gi
    cpu: 2000m
env:
  - name: PAPERLESS_TASK_WORKERS
    value: "2"
  - name: PAPERLESS_THREADS_PER_WORKER
    value: "2"

But this didn't explain the load average of 90.

Failure 2: The Control-Plane Was Scheduling App Workloads

When I checked where Paperless was running, it was on vm-srv-k3s-11 — the control-plane.

In a standard k3s setup, the control-plane has a node-role.kubernetes.io/control-plane:NoSchedule taint. User workloads shouldn't land there. But somewhere along the way, the Paperless deployment had picked up a toleration:

tolerations:
  - operator: Exists

operator: Exists with no key or effect matches every taint on every node, including NoSchedule on the control-plane. The pod scheduled there, and every OOMKill → restart cycle added another spike of CPU load to a node already running etcd, the k3s API server, CoreDNS, kube-proxy, and Longhorn replica management.

The fix was to remove the blanket toleration entirely. The Paperless deployment doesn't need to run on the control-plane.

With the toleration removed and the memory limit raised, load on vm-srv-k3s-11 dropped from 90 to 1.04 immediately. But two more problems had already developed in the background.

Failure 3: CoreDNS Was Generating 1.2 Million Queries Per Day

During the OOM cascade, I noticed AdGuard Home (running on two Raspberry Pi nodes in HA via Keepalived) was under unusually high load. I checked the query log: 1.2 million DNS queries in 24 hours for a three-node homelab cluster.

The culprit: CoreDNS default cache TTL.

CoreDNS ships with a 30-second cache TTL. Every pod that makes a DNS lookup for a Kubernetes service gets an answer that expires in 30 seconds. In a healthy cluster that's fine. During an OOM cascade — where pods are restarting constantly, new IPs are being assigned, and connection state is unstable — the DNS query rate explodes. Pods that are restarting frequently keep hammering CoreDNS for the same records.

The fix was a one-line patch to the CoreDNS ConfigMap:

kubectl patch configmap coredns -n kube-system --patch '
data:
  Corefile: |
    .:53 {
      errors
      health
      ready
      kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
      }
      prometheus :9153
      forward . /etc/resolv.conf
      cache 300
      loop
      reload
      loadbalance
    }
'

Raising the cache TTL from 30 to 300 seconds reduced the upstream query volume by roughly 10x. I also updated AdGuard Home (via Ansible) to enable optimistic caching and increase its cache size:

# ansible/roles/adguard/templates/AdGuardHome.yaml.j2
dns:
  cache_size: 67108864  # 64 MiB
  cache_optimistic: true

cache_optimistic: true means AdGuard returns the cached (possibly stale) answer immediately while refreshing in the background — eliminating the latency spike on cache expiry. Combined, these two changes brought the daily query count down to ~120k.

Failure 4: Worker Nodes Reporting Wrong Allocatable Memory

While fixing the above, I noticed something odd in kubectl describe node vm-srv-k3s-12:

Capacity:
  cpu:     4
  memory:  3981384Ki  ← ~3.8 GiB
Allocatable:
  cpu:     4
  memory:  3878584Ki  ← ~3.7 GiB

The VM was allocated 16 GiB in Proxmox. Why was kubelet reporting 3.8 GiB?

The answer is Proxmox balloon memory.

Balloon memory in Proxmox works like this: you set a dedicated (maximum) and a floating (minimum) value. When the host is under memory pressure, Proxmox can shrink the guest down to the floating minimum. The key detail: kubelet reads available memory at startup time. If kubelet starts when the VM has been ballooned down to its minimum, that's what it registers as the node's capacity — and it doesn't update that value dynamically.

My Terraform config had this:

memory {
  dedicated = 16384  # 16 GiB max
  floating  = 4096   # 4 GiB min ← too low
}

The workers had been under pressure during the OOM cascade, Proxmox had ballooned them down to 4 GiB, kubelet restarted and registered 3.8 GiB (4096 MB minus kernel + system overhead), and that's what Kubernetes thought the nodes had.

The fix: raise the minimum balloon to ensure kubelet always sees adequate memory:

memory {
  dedicated = 16384
  floating  = 8192   # 8 GiB min — safe floor for kubelet registration
}

After restarting k3s-agent on both workers, capacity showed correctly:

Capacity:
  memory: 16383272Ki  # 16 GiB

The Full Cascade, Traced

1Gi Paperless limit + Exists toleration
        ↓
OOMKill × 16 on the control-plane
        ↓
k3s-11 load average: 90
(etcd + API server + OCR workers + Longhorn replicas all competing)
        ↓
Pods restarting constantly → high DNS churn
        ↓
CoreDNS 30s TTL → 1.2M queries/day → AdGuard overload
        ↓
Balloon minimum 4096 MB → kubelet restart → 3.8 GiB registered
        ↓
Scheduler thinks workers have less capacity → over-schedules control-plane
        ↓
(back to top)

Each failure made the next one worse. Raising the memory limit without fixing the toleration would have helped Paperless but left the control-plane overloaded. Fixing the toleration without fixing the balloon minimum would have moved the problem to a worker node with 3.8 GiB of visible capacity. The DNS fix was independent but would have eventually caused its own stability issues at scale.

What Would Have Caught This Earlier

A few things would have surfaced these issues before they compounded:

1. Resource limit policy at admission time. A Kyverno require-resource-limits policy in Audit mode would have flagged the original 1 GiB limit as a potential issue and made it visible in PolicyReports before OOMKills started.

2. Control-plane taint monitoring. A simple alert on kube_pod_info{node="vm-srv-k3s-11"} unless kube_pod_info{namespace="kube-system"} would have fired the moment a user workload landed on the control-plane.

3. Node capacity validation in Terraform. The balloon minimum should be part of the VM definition review — ideally validated against the minimum kubelet requires to start safely.

None of these are exotic. They're standard practice in production clusters. The lesson is that homelab clusters accumulate the same failure modes as production clusters, just with less monitoring to catch them.

The Fixes, Summarised

Problem	Root Cause	Fix
OOMKill × 16	1 GiB limit too low for Tesseract burst	Limit → 3 GiB, workers → 2
Control-plane load 90	`tolerations: operator: Exists`	Remove blanket toleration
1.2M DNS queries/day	CoreDNS TTL 30s + OOM-induced restart churn	CoreDNS cache → 300s, AdGuard optimistic + 64 MiB
3.8 GiB allocatable	Proxmox balloon min 4096 MB, kubelet reads at startup	`floating = 8192` in Terraform

The cluster has been stable since. Paperless processes the same document batches without issue. CoreDNS query volume is down 90%. And kubelet now correctly reports 16 GiB on both workers.

The same failure modes — resource limits without ceiling analysis, overly permissive scheduling constraints, and hypervisor-level capacity mismatches — appear in enterprise Kubernetes deployments running on Azure VMs or bare-metal. The only difference is scale: one misconfigured limit in a 500-node cluster can trigger the same DNS storm, just with three extra zeros behind the query count.

External Secrets Operator + HashiCorp Vault: GitOps Secret Lifecycle in Kubernetes

david — Thu, 18 Jun 2026 19:08:06 +0000

Kubernetes Secrets are not secret.

base64 is not encryption. Anyone with kubectl get secret access can decode them instantly. Secrets stored in etcd are encrypted at rest only if you've explicitly configured encryption providers — and most clusters haven't. And if you're managing secrets in Git (even with SOPS or Sealed Secrets), the ciphertext is committed to version control forever.

The proper solution is an external secret store: a system specifically designed for secret storage, with access control, audit logging, and rotation built in. HashiCorp Vault is the most common choice. External Secrets Operator bridges Vault to Kubernetes — syncing secrets into the cluster without storing them in Git.

This post covers the full setup running on my k3s cluster: Vault deployment, bootstrap sequence, ClusterSecretStore, and the first real ExternalSecret.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Git (no secrets)
      ↓ ArgoCD syncs
  Vault (KV v2)          ←── You store secrets here
      ↑
External Secrets Operator
      ↓ creates/syncs
  k8s Secret (in-cluster, not in Git)
      ↓ consumed by
  Application Pod

The key property: nothing sensitive is ever committed to Git. ArgoCD manages all Kubernetes manifests except Secrets. Vault holds the actual values. ESO syncs them into the cluster on a refresh interval. Applications consume k8s.io/v1/Secret objects as normal — nothing changes from the application's perspective.

Step 1: Deploy Vault via ArgoCD

# kubernetes/system/vault/application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://helm.releases.hashicorp.com
    targetRevision: 0.28.1
    chart: vault
    helm:
      values: |
        server:
          standalone:
            enabled: true
          dataStorage:
            enabled: true
            size: 5Gi
            storageClass: nfs-client
        ui:
          enabled: true
          serviceType: ClusterIP
  destination:
    server: https://kubernetes.default.svc
    namespace: vault
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

NFS-backed storage for the Vault data directory. Vault runs as a single-node instance (standalone mode) — sufficient for a homelab, and it avoids the complexity of Raft consensus with multiple replicas.

Step 2: The Bootstrap Sequence

Vault ships sealed. Before it can serve any secrets, you must unseal it. This is a one-time manual operation — by design. Unsealing requires a quorum of key shares, so no single compromise can unlock the store.

# 1. Wait for the Vault pod to be running
kubectl wait --for=condition=Ready pod/vault-0 -n vault --timeout=120s

# 2. Initialize Vault (5 key shares, 3 required to unseal)
kubectl exec vault-0 -n vault -- vault operator init \
  -key-shares=5 \
  -key-threshold=3

# Output (save these — you cannot recover them):
# Unseal Key 1: abc...
# Unseal Key 2: def...
# Unseal Key 3: ghi...
# Unseal Key 4: jkl...
# Unseal Key 5: mno...
# Initial Root Token: hvs.xxx...

# 3. Unseal with 3 of the 5 keys
kubectl exec vault-0 -n vault -- vault operator unseal <key-1>
kubectl exec vault-0 -n vault -- vault operator unseal <key-2>
kubectl exec vault-0 -n vault -- vault operator unseal <key-3>

# 4. Verify unsealed
kubectl exec vault-0 -n vault -- vault status
# Sealed: false

Store the unseal keys and root token somewhere secure. Losing them means losing access to your Vault permanently. A password manager with hardware 2FA (Vaultwarden, 1Password, Bitwarden) works. Do not commit them to Git.

After a Vault pod restart (node reboot, update), you need to unseal again with 3 keys. Auto-unseal via AWS KMS or Azure Key Vault removes this manual step in production environments — acceptable for a homelab to skip.

Step 3: Enable KV v2 and Write Your First Secret

# Authenticate with the root token
kubectl exec -it vault-0 -n vault -- /bin/sh
vault login <root-token>

# Enable KV v2 at the 'secret/' path
vault secrets enable -path=secret kv-v2

# Write the first secret
vault kv put secret/authelia \
  hmac-secret="$(openssl rand -base64 32)"

# Verify
vault kv get secret/authelia

KV v2 maintains version history. You can roll back to a previous version of a secret, see who wrote what and when (with audit logging enabled), and compare versions. This is what makes Vault appropriate for compliance contexts — it's not just a secret store, it's a secret lifecycle management system.

Step 4: Deploy External Secrets Operator

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.external-secrets.io
    targetRevision: 0.10.4
    chart: external-secrets
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Step 5: Create the Token Secret and ClusterSecretStore

ESO needs a way to authenticate against Vault. The simplest approach for a single-cluster setup: a Kubernetes secret containing the Vault root token (or a scoped AppRole token for production).

# Create the token secret that ESO will use to authenticate against Vault
kubectl create secret generic vault-token \
  -n external-secrets \
  --from-literal=token=<vault-root-token>

Then the ClusterSecretStore:

# kubernetes/system/external-secrets/cluster-secret-store.yml
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault
spec:
  provider:
    vault:
      server: http://vault.vault.svc.cluster.local:8200
      path: secret
      version: v2
      auth:
        tokenSecretRef:
          name: vault-token
          namespace: external-secrets
          key: token

ClusterSecretStore (vs SecretStore) is cluster-scoped — any namespace can reference it. For multi-tenant clusters where namespaces shouldn't cross-read each other's secrets, use namespace-scoped SecretStore instead.

The path: secret and version: v2 match the KV mount we created in step 3.

Step 6: The First ExternalSecret

# kubernetes/apps/authelia/external-secret.yml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: authelia-secrets
  namespace: apps
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault
    kind: ClusterSecretStore
  target:
    name: authelia-secrets
    creationPolicy: Merge
  data:
    - secretKey: hmac-secret
      remoteRef:
        key: secret/authelia
        property: hmac-secret

Three things worth noting:

refreshInterval: 1h — ESO re-reads from Vault every hour. If you rotate the secret in Vault, the k8s Secret is updated within an hour. No pod restart required for most applications that read secrets from mounted files (as opposed to environment variables, which require a restart).

creationPolicy: Merge — Instead of creating a new Secret from scratch, ESO merges the Vault-sourced key into an existing Secret. This is useful when a Secret needs some values from Vault (sensitive) and others from a ConfigMap (non-sensitive). The application sees a single unified Secret.

remoteRef.key — The full Vault path is secret/data/authelia (KV v2 prepends data/), but ESO handles the /data/ prefix automatically when version: v2 is set. You write secret/authelia in the ExternalSecret.

Verifying It Works

# Check the ExternalSecret sync status
kubectl get externalsecret authelia-secrets -n apps

# Output:
# NAME               STORE   REFRESH INTERVAL   STATUS         READY
# authelia-secrets   vault   1h                 SecretSynced   True

# Check the resulting k8s Secret
kubectl get secret authelia-secrets -n apps -o jsonpath='{.data.hmac-secret}' | base64 -d

If STATUS shows SecretSyncedError, check:

kubectl describe externalsecret authelia-secrets -n apps for the error message
Vault pod is running and unsealed (kubectl exec vault-0 -n vault -- vault status)
The token secret exists in the external-secrets namespace
The KV path actually exists in Vault (vault kv get secret/authelia)

What You Get

Audit log: Every secret read from Vault is logged. vault audit enable file file_path=/vault/logs/audit.log gives you a full trail of who (which token) accessed what secret and when.
Rotation without redeployment: Rotate a secret in Vault, ESO syncs it within the refresh interval. For file-mounted secrets, the pod picks it up without restart.
No secrets in Git: The ExternalSecret manifest commits to Git. It describes what to sync and where from — but not the value. The value stays in Vault.
Compliance evidence: KV v2 version history + audit log gives you the access evidence ISO 27001 (A.9.4 — System and Application Access Control) and NIS2 require.

Next: AppRole Authentication

The setup above uses the Vault root token for ESO authentication. That works, but the root token has unrestricted access to everything in Vault.

For a more hardened setup, create a Vault AppRole with a policy scoped to only the secrets ESO needs:

# Policy: ESO can only read under secret/data/
vault policy write eso-readonly - <<EOF
path "secret/data/*" {
  capabilities = ["read"]
}
EOF

# AppRole
vault auth enable approle
vault write auth/approle/role/eso \
  policies="eso-readonly" \
  token_ttl=1h \
  token_max_ttl=4h

# Get role_id and secret_id for ESO
vault read auth/approle/role/eso/role-id
vault write -f auth/approle/role/eso/secret-id

Update the ClusterSecretStore to use AppRole auth instead of tokenSecretRef. This follows the principle of least privilege — a compromise of the ESO token only exposes read access to secrets, not root-level Vault control.

Full Observability on k3s: kube-prometheus-stack + Loki + Grafana OIDC

david — Sun, 14 Jun 2026 12:07:33 +0000

A Kubernetes cluster without observability is a black box. You deploy services, they run — until they don't. When something breaks at 2am, you need metrics, logs, and alerts that actually tell you what happened.

This is the full observability stack running on my bare-metal k3s cluster: kube-prometheus-stack for metrics and alerting, Loki with Garage S3 for log persistence, Promtail collecting logs from non-Kubernetes nodes via Ansible, SNMP metrics from the MikroTik router, and Grafana with Authelia OIDC — so there's one login for everything.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Metrics                          Logs
───────                          ────
kube-prometheus-stack            Promtail (k8s DaemonSet)
  ├── Prometheus (30d retention)   ├── All pod logs
  ├── Alertmanager                 └── System logs
  └── node-exporter (k8s)
                                 Promtail (Ansible, bare-metal)
SNMP Exporter                      ├── /var/log/syslog
  └── MikroTik RB5009 → Prometheus └── /var/log/auth.log
                                   └── Docker container logs
node_exporter (Ansible)
  └── RPi + LXC nodes → Prometheus

                    Grafana (OIDC via Authelia)
                         │
                    ┌────┴────┐
               Prometheus    Loki → Garage S3

Everything lands in Grafana. One URL, one SSO login, metrics and logs side by side.

Step 1: kube-prometheus-stack via ArgoCD

The kube-prometheus-stack Helm chart installs Prometheus, Alertmanager, Grafana, and all the associated CRDs in a single deployment.

# kubernetes/system/monitoring/kube-prometheus-stack.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kube-prometheus-stack
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://prometheus-community.github.io/helm-charts
    targetRevision: 61.3.2
    chart: kube-prometheus-stack
    helm:
      values: |
        prometheusOperator:
          crds:
            enabled: false      # manage CRDs separately to avoid ArgoCD ordering issues

        prometheus:
          prometheusSpec:
            retention: 30d
            dnsConfig:
              options:
                - name: ndots
                  value: "1"
            storageSpec:
              volumeClaimTemplate:
                spec:
                  storageClassName: longhorn
                  accessModes: ["ReadWriteOnce"]
                  resources:
                    requests:
                      storage: 20Gi

        grafana:
          enabled: true
          sidecar:
            datasources:
              enabled: true
              searchNamespace: ALL
            dashboards:
              enabled: true
              searchNamespace: ALL
              label: grafana_dashboard
              labelValue: "1"
          additionalDataSources:
            - name: Loki
              type: loki
              access: proxy
              url: http://loki.monitoring.svc.cluster.local:3100
              jsonData:
                maxLines: 1000

prometheusOperator.crds.enabled: false — CRDs and the operator have an ordering dependency. Disabling CRD installation here and managing them separately prevents ArgoCD sync failures on fresh installs.

retention: 30d with Longhorn storage — Prometheus data persists across pod restarts and node reboots. Without storageSpec, metrics live only in the pod's ephemeral storage.

ndots: 1 — reduces DNS lookup latency inside the cluster. With Kubernetes' default of 5, every single-label hostname triggers 5 NXDOMAIN lookups before resolution. Setting it to 1 cuts that overhead significantly for monitoring scrapes.

Step 2: Grafana OIDC via Authelia

Grafana's generic OAuth provider connects to Authelia. Users authenticate once — the same session covers Grafana, Vaultwarden, and every other protected service.

        grafana:
          grafana.ini:
            server:
              domain: monitoring.yourdomain.com
              root_url: https://monitoring.yourdomain.com

            auth:
              oauth_auto_login: true

            auth.generic_oauth:
              enabled: true
              name: Authelia
              client_id: grafana
              client_secret: "<your-oidc-client-secret>"
              scopes: openid profile email groups
              auth_url: https://auth.yourdomain.com/api/oidc/authorization
              token_url: https://auth.yourdomain.com/api/oidc/token
              api_url: https://auth.yourdomain.com/api/oidc/userinfo
              login_attribute_path: preferred_username
              groups_attribute_path: groups
              role_attribute_path: >
                contains(groups[*], 'admins') && 'Admin' || 'Viewer'

role_attribute_path maps Authelia group membership to Grafana roles using JMESPath. Members of the admins group get Admin access; everyone else gets read-only Viewer access. No per-user role management in Grafana.

In Authelia, add the client:

identity_providers:
  oidc:
    clients:
      - client_id: grafana
        client_name: Grafana
        client_secret: "<bcrypt-hash>"
        authorization_policy: one_factor
        redirect_uris:
          - https://monitoring.yourdomain.com/login/generic_oauth
        scopes:
          - openid
          - profile
          - email
          - groups

Step 3: Loki with Garage S3 Storage

Loki needs durable object storage. Storing logs on a local PVC means a node failure loses your log history. Garage — the same lightweight S3 instance already deployed for Velero backups — handles this.

# kubernetes/system/monitoring/loki.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: loki
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://grafana.github.io/helm-charts
    targetRevision: 6.6.2
    chart: loki
    helm:
      values: |
        deploymentMode: SingleBinary

        loki:
          auth_enabled: false
          commonConfig:
            replication_factor: 1

          storage:
            type: s3
            bucketNames:
              chunks: loki-data
              ruler: loki-data
              admin: loki-data
            s3:
              endpoint: http://garage.apps.svc.cluster.local:3900
              region: homelab
              s3ForcePathStyle: true
              insecure: true

          limits_config:
            retention_period: 30d
            ingestion_rate_mb: 2
            ingestion_burst_size_mb: 4

          compactor:
            retention_enabled: true
            compaction_interval: 10m
            retention_delete_delay: 2h

          schemaConfig:
            configs:
              - from: "2024-04-01"
                object_store: s3
                store: tsdb
                schema: v13
                index:
                  prefix: index_
                  period: 24h

        singleBinary:
          replicas: 1
          extraEnv:
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: loki-s3-secrets
                  key: access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: loki-s3-secrets
                  key: secret-access-key

        # disable unused replicated components
        read:
          replicas: 0
        write:
          replicas: 0
        backend:
          replicas: 0

        chunksCache:
          allocatedMemory: 512
        resultsCache:
          allocatedMemory: 512

        lokiCanary:
          enabled: false
        test:
          enabled: false

s3ForcePathStyle: true and insecure: true (plain HTTP to the cluster-internal Garage endpoint) are both required. Garage uses path-style URLs, and the internal cluster DNS endpoint is HTTP — Loki's TLS verification would fail on a self-signed cert.

Before deploying, create the Loki bucket in Garage:

kubectl exec -it -n apps deploy/garage -- /garage bucket create loki-data
kubectl exec -it -n apps deploy/garage -- /garage key create loki-key
kubectl exec -it -n apps deploy/garage -- \
  /garage bucket allow loki-data --read --write --owner --key loki-key

Then create the credentials secret:

apiVersion: v1
kind: Secret
metadata:
  name: loki-s3-secrets
  namespace: monitoring
type: Opaque
stringData:
  access-key-id: "<garage-key-id>"
  secret-access-key: "<garage-secret-key>"

Step 4: Promtail as a DaemonSet

Promtail ships with the Loki chart and runs as a DaemonSet — one pod per node — collecting all container logs automatically:

# kubernetes/system/monitoring/promtail.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: promtail
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://grafana.github.io/helm-charts
    targetRevision: 6.16.4
    chart: promtail
    helm:
      values: |
        config:
          clients:
            - url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push

Promtail discovers all pods automatically via the Kubernetes API. No per-service configuration needed.

Step 5: node_exporter + Promtail on Bare-Metal Nodes

The Raspberry Pi nodes and Docker LXC container are not part of the k3s cluster — they need the monitoring agent installed via Ansible.

The monitoring_agent role deploys node_exporter and Promtail as Docker containers:

# ansible/roles/monitoring_agent/tasks/main.yml
- name: Deploy node_exporter
  ansible.builtin.template:
    src: docker-compose.yml.j2
    dest: /opt/docker/node_exporter/docker-compose.yml

- name: Deploy promtail configuration
  ansible.builtin.template:
    src: promtail.yml.j2
    dest: /opt/docker/promtail/promtail.yml
  notify: Restart promtail

The node_exporter Compose template exposes system metrics on port 9100:

services:
  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    restart: unless-stopped
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)'
    ports:
      - "9100:9100"

The Promtail config ships system logs, auth logs, and Docker container logs to Loki:

clients:
  - url: http://{{ monitoring_core_host }}:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          host: {{ inventory_hostname }}
          __path__: /var/log/syslog

  - job_name: auth
    static_configs:
      - targets: [localhost]
        labels:
          host: {{ inventory_hostname }}
          __path__: /var/log/auth.log

  - job_name: docker
    static_configs:
      - targets: [localhost]
        labels:
          host: {{ inventory_hostname }}
          __path__: /var/lib/docker/containers/*/*-json.log
    pipeline_stages:
      - json:
          expressions:
            output: log
            stream: stream
            container: attrs.name
      - labels:
          stream:
          container:
      - output:
          source: output

Step 6: SNMP Monitoring for MikroTik

The MikroTik router runs SNMP but doesn't expose Prometheus metrics natively. The SNMP exporter bridges this gap — it scrapes the router via SNMP and translates the results to Prometheus format.

In the Prometheus config (inside kube-prometheus-stack's additionalScrapeConfigs):

scrape_configs:
  - job_name: 'mikrotik_snmp'
    static_configs:
      - targets:
          - '10.0.10.1'     # MikroTik management IP
    metrics_path: /snmp
    params:
      module: [if_mib]
      auth: [public_v2]
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: snmp-exporter:9116

This gives you per-interface traffic counters, error rates, and operational status for every port on the switch — all visible in Grafana.

Step 7: Custom Dashboard as ConfigMap

Grafana's sidecar watches for ConfigMaps with the label grafana_dashboard: "1" and automatically imports them. This means dashboards are version-controlled in Git:

# kubernetes/system/monitoring/loki-dashboard.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: loki-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  loki-logs.json: |
    {
      "title": "Loki: Kubernetes Logs",
      "uid": "loki-kubernetes-logs",
      "panels": [
        {
          "title": "Log Stream",
          "type": "logs",
          "targets": [
            {
              "expr": "{namespace=~\"$namespace\", pod=~\"$pod\"}"
            }
          ]
        }
      ]
    }

No manual dashboard import, no "save to disk" issues after container restarts. The dashboard JSON lives in Git, ArgoCD applies it, the sidecar picks it up automatically.

The Result

After deploying all components:

Metrics: Prometheus scrapes every k3s node, every bare-metal node, and the MikroTik router. 30 days of history on Longhorn.
Logs: All pod logs and system logs from every node flow into Loki, stored durably on Garage S3. 30 day retention with automatic compaction.
Dashboards: Grafana shows metrics and logs in a single view, with Loki datasource pre-configured. Custom dashboards deploy via ArgoCD with zero manual steps.
Access: One SSO login via Authelia. Group membership determines the Grafana role.
Storage efficiency: Garage serves both Velero backups and Loki log chunks from a single lightweight deployment — two use cases, one binary.

The same three-layer observability model — metrics, logs, traces — applies in enterprise Azure environments with Azure Monitor, Log Analytics, and Application Insights. If you're building the network foundation that those services sit on, the Enterprise Terraform Blueprints cover the Private Link isolation layer for Azure monitoring endpoints.

HA DNS for Homelab: Unbound + AdGuard Home + Keepalived on Raspberry Pi

david — Sun, 14 Jun 2026 12:06:45 +0000

DNS is the most critical service in any network. If it goes down, nothing works — browsers can't resolve hostnames, services can't reach each other, and the error messages are uniformly unhelpful. In a homelab, a single DNS server is a single point of failure.

This is the DNS architecture running on two Raspberry Pi 4B edge nodes in my homelab: Unbound as a recursive resolver, AdGuard Home for filtering, and Keepalived for automatic failover. The whole stack is managed with Ansible.

View the complete homelab infrastructure source on GitHub 🐙

The Architecture

Client (any device on the network)
        │
        ▼
Virtual IP 10.0.20.5 (Keepalived VIP)
        │
        ├── Primary: rpi-srv-01 (10.0.20.2) — MASTER
        └── Backup:  rpi-srv-02 (10.0.20.3) — BACKUP
               │
               ▼
        AdGuard Home (filtering + blocking)
               │
               ▼
        Unbound :5335 (recursive resolver)
               │
               ▼
        Root DNS servers (no upstream forwarder)

Clients point to a single IP (the VIP). If the primary Pi fails, Keepalived moves the VIP to the backup node within seconds. No client reconfiguration, no DNS TTL wait.

Why Recursive Resolution

Most homelab DNS setups forward queries to a public upstream (Cloudflare, Google, Quad9). That works, but every query you make is visible to a third party.

Unbound resolves queries by starting at the DNS root servers and following delegations down — the same way authoritative DNS actually works. No single upstream sees your full query history. The trade-off is slightly higher first-query latency; subsequent queries are cached locally.

Step 1: Unbound Ansible Role

Unbound runs in Docker on each Pi. The Ansible role deploys the Compose stack and configuration:

# ansible/roles/unbound/tasks/main.yml
- name: Create Unbound configuration directory
  ansible.builtin.file:
    path: /opt/unbound/conf
    state: directory
    mode: '0755'

- name: Deploy Unbound configuration
  ansible.builtin.template:
    src: unbound.conf.j2
    dest: /opt/unbound/conf/unbound.conf
  notify: Restart Unbound

- name: Deploy Unbound Docker Compose stack
  ansible.builtin.copy:
    dest: /opt/unbound/docker-compose.yml
    content: |
      services:
        unbound:
          image: klutchell/unbound:latest
          container_name: unbound
          restart: unless-stopped
          ports:
            - "5335:53/udp"
            - "5335:53/tcp"
          healthcheck:
            test: ["CMD", "dig", "+short", "@127.0.0.1", "-p", "5335", "google.com"]
            interval: 30s
            timeout: 10s
            retries: 3
          labels:
            - "autoheal=true"
          volumes:
            - /opt/unbound/conf/unbound.conf:/etc/unbound/unbound.conf

        autoheal:
          image: willfarrell/autoheal:latest
          container_name: autoheal
          restart: always
          environment:
            - AUTOHEAL_CONTAINER_LABEL=autoheal
          volumes:
            - /var/run/docker.sock:/var/run/docker.sock

- name: Optimize kernel network buffers for Unbound
  ansible.posix.sysctl:
    name: "{{ item.name }}"
    value: "{{ item.value }}"
    state: present
    sysctl_set: true
  loop:
    - { name: "net.core.rmem_max", value: "4194304" }
    - { name: "net.core.wmem_max", value: "4194304" }

Two things worth noting:

Port 5335 — Unbound does not bind to port 53. That port belongs to AdGuard Home. AdGuard forwards to 127.0.0.1:5335.

Autoheal — watches for containers with the autoheal=true label and restarts them if the healthcheck fails. DNS downtime on a Pi is silent and annoying; autoheal catches it automatically.

The Unbound configuration template:

# ansible/roles/unbound/templates/unbound.conf.j2
server:
    interface: 0.0.0.0
    port: 53
    do-ip4: yes
    do-udp: yes
    do-tcp: yes
    do-ip6: no

    access-control: 127.0.0.0/8 allow
    access-control: 10.0.0.0/8 allow

    hide-identity: yes
    hide-version: yes

    # Cache settings
    cache-min-ttl: 60
    cache-max-ttl: 86400
    prefetch: yes

    # DNSSEC
    auto-trust-anchor-file: "/var/lib/unbound/root.key"

forward-zone:
    name: "."
    forward-first: no

access-control: 10.0.0.0/8 allow permits queries from all homelab VLANs. Queries from outside that range are refused.

Step 2: AdGuard Home Ansible Role

AdGuard runs in network_mode: host so it can bind to port 53 directly on the Pi's interface:

# ansible/roles/adguard/tasks/main.yml
- name: Deploy AdGuard Home Docker Compose file
  ansible.builtin.copy:
    dest: /opt/adguardhome/docker-compose.yml
    content: |
      services:
        adguardhome:
          image: adguard/adguardhome
          container_name: adguardhome
          restart: always
          network_mode: host
          volumes:
            - /opt/adguardhome/work:/opt/adguardhome/work
            - /opt/adguardhome/conf:/opt/adguardhome/conf

In the AdGuard UI, set the upstream DNS to 127.0.0.1:5335 (Unbound). All filtered queries that pass through AdGuard's blocklists are forwarded to Unbound for recursive resolution.

Step 3: Config Sync to the Replica

AdGuard doesn't natively replicate configuration between instances. The role uses adguardhome-sync on the backup node to pull config from the primary:

- name: Deploy AdGuardHome-Sync on replica node
  when: inventory_hostname == 'rpi-srv-02'
  block:
    - name: Deploy Sync Docker Compose
      ansible.builtin.copy:
        dest: /opt/adguardhome-sync/docker-compose.yml
        content: |
          services:
            adguardhome-sync:
              image: ghcr.io/bakito/adguardhome-sync
              container_name: adguardhome-sync
              restart: unless-stopped
              environment:
                - ORIGIN_URL=http://10.0.20.2:3001
                - ORIGIN_USERNAME=dw
                - ORIGIN_PASSWORD={{ vault_adguard_password }}
                - REPLICA1_URL=http://127.0.0.1:3001
                - REPLICA1_USERNAME=dw
                - REPLICA1_PASSWORD={{ vault_adguard_password }}
                - CRON=*/10 * * * *

Every 10 minutes, the replica pulls filter lists, custom rules, and settings from the primary. If the primary goes down, the replica is already up to date and takes over immediately.

Step 4: Keepalived for Failover

Keepalived uses VRRP to maintain a shared Virtual IP across both nodes:

# ansible/roles/keepalived/templates/keepalived.conf.j2
vrrp_instance VI_1 {
    state {{ "MASTER" if inventory_hostname == 'rpi-srv-01' else "BACKUP" }}
    interface eth0
    virtual_router_id 51
    priority {{ 150 if inventory_hostname == 'rpi-srv-01' else 100 }}
    advert_int 1

    authentication {
        auth_type PASS
        auth_pass {{ keepalived_auth_pass }}
    }

    virtual_ipaddress {
        10.0.20.5/24
    }
}

The primary node (rpi-srv-01) has priority 150, the backup has 100. As long as the primary is up, it holds the VIP. If it stops sending VRRP advertisements, the backup promotes itself and takes over the IP within ~3 seconds.

VRRP uses multicast. If your switch filters multicast between ports (MikroTik does by default), you need to permit 224.0.0.18 on the VLAN carrying the Pi nodes.

The Ansible task:

# ansible/roles/keepalived/tasks/main.yml
- name: Install Keepalived
  ansible.builtin.apt:
    name: keepalived
    state: present

- name: Deploy Keepalived configuration from template
  ansible.builtin.template:
    src: keepalived.conf.j2
    dest: /etc/keepalived/keepalived.conf
  notify: Restart Keepalived

- name: Ensure Keepalived service is started and enabled
  ansible.builtin.service:
    name: keepalived
    state: started
    enabled: true

Deploying with Ansible

The three roles are applied to the rpi_nodes host group:

# ansible/playbooks/site.yml (relevant section)
- hosts: rpi_nodes
  roles:
    - common
    - docker
    - unbound
    - adguard
    - keepalived

# Deploy to both Pi nodes
ansible-playbook ansible/playbooks/site.yml --limit rpi_nodes

# Dry run first
ansible-playbook ansible/playbooks/site.yml --limit rpi_nodes --check

Testing Failover

Confirm the VIP is on the primary:

ip addr show eth0 | grep 10.0.20.5
# Should show the VIP on rpi-srv-01 only

Simulate a failure:

# On rpi-srv-01
sudo systemctl stop keepalived

# On any client
dig @10.0.20.5 google.com
# Should still resolve — now via rpi-srv-02

Check which node now holds the VIP:

# On rpi-srv-02
ip addr show eth0 | grep 10.0.20.5
# VIP should now appear here

Restart Keepalived on the primary and it re-claims the VIP automatically.

The Result

All devices point to 10.0.20.5 — a single address that never changes
Queries are filtered by AdGuard (blocklists, custom rules) then resolved recursively by Unbound
If either Pi goes down, the other takes over within 3 seconds
Filter lists and config sync automatically every 10 minutes
Kernel buffer tuning ensures Unbound can handle high-volume UDP traffic without dropping queries

The entire stack is idempotent Ansible — running the playbook again changes nothing if everything is already in the desired state.

DNS control is the foundation of any Zero-Trust network — on-premises or in the cloud. In Azure, the equivalent of this setup is Azure Private DNS Zones with Private Link resolvers. The Enterprise Terraform Blueprints include pre-configured Private DNS Zones for all major Azure PaaS services.

DEV Community: david

SLO Burn-Rate Alerting with Prometheus: Beyond Threshold Alerts

Error Budgets, Briefly

The SLI: HTTP Probe Success Rate

Recording Rules: Pre-Computing the Windows

The Alert Rules

Fast Burn: Page Immediately

Slow Burn: Create a Ticket

Comparing Against Threshold Alerting

Deploying as a PrometheusRule

What the Error Budget Dashboard Shows

Limitations

I Hardened Pod securityContext and Broke 9 Containers in Production

The Two Wrong Assumptions

Catching It: Why "Application: Synced/Healthy" Lied

The Logs Told the Real Story Every Time

The Recovery Trap: selfHeal Undoes Manual Fixes

The Fix, Applied Selectively

The Lesson

Hardening Unattended Raspberry Pi Edge Nodes: Watchdog, fail2ban, nftables, and the Mistakes That Take Down DNS

Why "It's Just DNS" Needs More Hardening, Not Less

Hardware Watchdog: Recovering From a Hang Software Can't See

fail2ban: The One Exposed Surface

The nftables Trap: Don't Touch /etc/nftables.conf

Stopping Slow SD-Card Death

Memory Limits: Catching a Leak Before It Takes the Whole Pi Down

Local Config Backup: The Gap Nobody Noticed

Alerting That Survives the Main Alerting Stack Being Down

What Actually Got Tested, Not Just Written

IPv6 NAT66 Behind a FritzBox: The RouterOS 7 Bug That Broke WiFi Clients

The Topology

Step 1: ULA Addresses Per VLAN

Step 2: Accept RA on ether1

Step 3: NAT66 Masquerade

Step 4: IPv6 Firewall

The Bug: RouterOS 7 Sends RA on All Interfaces

ULA vs. GUA: Why Not Just Use the ISP Prefix?

Verifying the Setup

My Firewall Had 77 Rules. Terraform Knew About 22 of Them.

How You End Up With Four Generations of the Same Firewall

The Bug This Actually Caused

Finding the Actual Scope of the Problem

Deleting Firewall Rules Without Locking Yourself Out

What's Left

The General Lesson

Kyverno: Supply Chain Security as Admission Control on Kubernetes

Why Admission Control

Installing Kyverno via ArgoCD

Policy 1: Require Resource Limits (Audit)

Policy 2: Disallow Privileged Containers (Enforce)

Policy 3: Disallow :latest Image Tag (Audit)

Real-World Example: The Authelia :latest Violation

The Audit → Enforce Rollout Strategy

Scoping Policies to Specific Namespaces

PolicyReport: The Visibility Layer

Enterprise Bridge

I Ran Gitleaks Against My Own Repo and Found 12 Real Secrets

What Gitleaks Found

Why a Private Repo Doesn't Make This Fine

Setting Up Gitleaks Without Getting Blocked by History

The Three-Layer Hook Setup

Allowlisting Real False Positives

The Remediation Plan

Confirming Remediation Actually Worked

What This Doesn't Fix

ArgoCD Gotchas: Cache Staleness and the SharedResourceWarning Nobody Explains

The Symptom

Why This Happens: selfHeal Plus a Stale Cache

The Fix: Force a Hard Refresh

The StatefulSet Exception

A Second, Different-Looking Bug With a Related Cause: SharedResourceWarning

Telling the Two Apart

How a 1 GiB Memory Limit Took Down My Entire k3s Cluster

The Setup

Failure 1: Paperless OOMKilled 16 Times in 5 Hours

Failure 2: The Control-Plane Was Scheduling App Workloads

Failure 3: CoreDNS Was Generating 1.2 Million Queries Per Day

Failure 4: Worker Nodes Reporting Wrong Allocatable Memory

The Full Cascade, Traced

What Would Have Caught This Earlier

Policy 3: Disallow `:latest` Image Tag (Audit)

Real-World Example: The Authelia `:latest` Violation

Why This Happens: `selfHeal` Plus a Stale Cache