DEV Community: david

Gitea Actions on k3s: Self-Hosted CI Without GitHub

david — Thu, 12 Nov 2026 16:44:49 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

GitHub Actions is free for public repos. For private repos, you get 2,000 minutes/month on the free tier. A homelab infrastructure repo that runs Terraform plan, Ansible lint, and container builds on every PR burns through 2,000 minutes in about a week. Gitea Actions runs the same YAML workflows on your own hardware, and the BMAX Mini PC* with 64GB RAM has more than enough for a Gitea instance and a handful of runners.

View the complete homelab infrastructure source on GitHub

Gitea Actions: The GitHub Actions Clone

Gitea added Actions in 1.19 as an experimental feature and has enabled it by default since 1.21. It's a CI system that reads workflow files in the same YAML format as GitHub Actions. The key differences:

Workflows go in .gitea/workflows/; Gitea also reads .github/workflows/ when that's all a repo has
The runner is a separate binary (act_runner) that polls Gitea for jobs
Logs and artifacts go to Gitea's own storage (local disk by default, configurable to MinIO/S3)
Secrets and variables are managed in the Gitea UI

The YAML syntax is the same, and actions like actions/checkout@v4 are pulled from github.com by default. Most GitHub Actions workflows work in Gitea with minimal changes.

Deploying Gitea on k3s

Gitea runs as a StatefulSet with PostgreSQL:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: gitea
  namespace: gitea
spec:
  replicas: 1
  serviceName: gitea
  selector:
    matchLabels:
      app: gitea
  template:
    metadata:
      labels:
        app: gitea
    spec:
      containers:
        - name: gitea
          image: gitea/gitea:1.22
          ports:
            - containerPort: 3000
            - containerPort: 22
          env:
            - name: GITEA__DATABASE__DB_TYPE
              value: "postgres"
            - name: GITEA__DATABASE__HOST
              value: "gitea-db.gitea.svc:5432"
            - name: GITEA__DATABASE__NAME
              value: "gitea"
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 10Gi

The Action Runner

The act_runner binary runs as a Deployment, polling Gitea for jobs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitea-runner
  namespace: gitea
spec:
  replicas: 2
  selector:
    matchLabels:
      app: gitea-runner
  template:
    metadata:
      labels:
        app: gitea-runner
    spec:
      containers:
        - name: runner
          image: gitea/act_runner:latest
          env:
            - name: GITEA_INSTANCE_URL
              value: "http://gitea.gitea.svc:3000"
            - name: GITEA_RUNNER_REGISTRATION_TOKEN
              valueFrom:
                secretKeyRef:
                  name: gitea-runner-token
                  key: token
            - name: GITEA_RUNNER_NAME
              value: "k3s-runner"
            - name: GITEA_RUNNER_LABELS
              value: "ubuntu-latest:docker://node:20-bookworm"

The runner registration token is generated in the Gitea UI under Site Administration → Actions → Runners → Create new Runner.

Two caveats with this manifest. With docker:// labels, act_runner starts each job in a container, so it needs a Docker daemon; on k3s that means a Docker-in-Docker sidecar or the act_runner dind image, which runs privileged. And act_runner stores its registration in a .runner file, so give each replica persistent storage (a StatefulSet fits better than a Deployment here), or every restarted pod registers as a new runner.

Workflow Migration

A GitHub Actions workflow migrates to Gitea by moving the file, or by leaving it where it is if the repo has no .gitea/workflows/ directory:

.github/workflows/ci.yml  →  .gitea/workflows/ci.yml

The YAML content stays the same. Gitea provides the github context for compatibility, so github.event_name, github.repository and github.sha keep working; gitea.* is available as an alias.

Docker-in-Docker

Many CI workflows need Docker (building images, running containers). The Gitea runner supports this either by running job containers privileged or by mounting the Docker socket into them:

# .gitea/workflows/docker-build.yml
name: Build and Push
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build
        run: docker build -t myapp:${{ github.sha }} .
      - name: Push
        run: |
          echo ${{ secrets.DOCKER_PASSWORD }} | docker login -u ${{ secrets.DOCKER_USERNAME }} --password-stdin
          docker push myapp:${{ github.sha }}

The runner label ubuntu-latest:docker://node:20-bookworm tells the runner to execute jobs inside a node:20-bookworm container, which has no Docker CLI. For image builds, point the label at an image that ships one, such as catthehacker's act image, and enable container.privileged: true (or a socket mount via container.options) in the runner's config.yaml. Privileged mode is not a label setting:

GITEA_RUNNER_LABELS: "ubuntu-latest:docker://ghcr.io/catthehacker/ubuntu:act-latest"

Resource Limits

On a homelab BMAX Mini PC*, the Gitea runner shouldn't eat all the resources. Set resource limits on the runner Deployment:

resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "2"
    memory: "2Gi"

Two runners capped at 2 GiB each is 4 GiB at most, negligible on a 64GB machine. The runners share CPU with the other k3s workloads: the scheduler reserves only the 500m request, and cpu: "2" is a throttling ceiling, not a reservation. Note that with Docker-in-Docker the job containers run inside the dind sidecar, so set limits on that container too.

The Trade-off

Gitea Actions is not as mature as GitHub Actions. Matrix builds and actions from the GitHub marketplace work, but some workflow syntax is ignored or only partly supported: jobs.<job_id>.environment, problem matchers, error annotations, complex runs-on expressions, and GitHub-only permissions scopes such as id-token. For a homelab infrastructure repo that runs Terraform, Ansible, and container builds, those gaps don't matter. Terraform changes still go through Atlantis for the actual apply, so Gitea Actions only needs to handle lint and plan. The YAML syntax is the same, the runner runs on your hardware, and there's no minute limit.

Raspberry Pi 5 Alternatives Compared 2026

david — Tue, 10 Nov 2026 15:38:55 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

Every time Raspberry Pi stock tightens or prices creep up, the same question comes back: is there a better SBC for homelab edge work in 2026? I run Raspberry Pi 4B units for DNS and Keepalived in my own rack, hardened per my watchdog/fail2ban/nftables setup, and I'd recommend the Pi 5 to anyone building that same role today. This is the comparison I'd want before spending money on an alternative: what the competing boards offer, where they win, and where the Pi's ecosystem still makes it the safer default.

View the complete homelab infrastructure source on GitHub

TL;DR: The Picks

Model	SoC	RAM options	Best For
Raspberry Pi 5 (8GB) Starter Kit*	Broadcom BCM2712 (4x Cortex-A76)	1/2/4/8/16GB	Best Overall / Ecosystem
Radxa Rock 5B	Rockchip RK3588 (4x A76 + 4x A55)	4/8/16GB	Best Raw Performance
Orange Pi 5 Plus	Rockchip RK3588 (4x A76 + 4x A55)	4/8/16/32GB	Best Specs-per-Euro
Beelink S12 Pro mini PC*	Intel N100	16GB DDR4 (SO-DIMM)	Best "Just Skip SBCs" Option

SBC prices have moved repeatedly since 2025, mostly because of memory costs, so I've left them out. Check the current listing for the RAM tier you actually need.

The short version: the Raspberry Pi 5 remains the right default for anything that needs to work with the widest software and community support: DNS, Keepalived, Home Assistant, PiKVM-style appliances. If the workload is CPU- or RAM-bound (local inference, heavier containerized services, multiple VMs via KVM), the RK3588 boards (Rock 5B or Orange Pi 5 Plus) offer more silicon for the money. And if the form factor doesn't need to be SBC-sized, a budget N100 mini PC often beats all of them on day-to-day reliability.

Why This Comparison Exists at All

The Pi 5 is a good board. People search for "alternatives" because of availability and price, not a spec deficiency. Raspberry Pi's retail pricing has crept up since the Pi 4 era, and RK3588 boards from Radxa, Orange Pi, and others outperform it on paper for similar money. Whether that advantage survives real homelab use is the question.

Selection Criteria for Homelab Edge Use

Software and driver maturity. This is where the Pi wins before specs enter the conversation. Raspberry Pi OS, a kernel Raspberry Pi maintains itself, and a decade of community documentation mean almost everything works, including obscure GPIO/HAT hardware. RK3588 boards run mainline Linux reasonably well in 2026, but VPU/NPU acceleration, some peripheral drivers, and firmware still lag behind. Vendor kernel forks and less-tested distro images are common.

Actual (not marketed) idle power draw. For anything running 24/7 on a homelab shelf, idle wattage matters more than peak benchmark numbers.

Long-term software support. Raspberry Pi states the Pi 5 will stay in production until at least January 2036. Smaller board vendors support their boards too, but for shorter and less predictable periods. That matters if you don't want to re-flash and re-provision in two years.

Networking. The Pi 5 has a single 1GbE port. Both RK3588 boards here ship 2.5GbE, which matters if the device does anything network-intensive (routing, NAS-adjacent duties).

Price at the RAM tier you need. SBC pricing is not linear across RAM tiers, and the advertised starting price of any board is the smallest variant, not the 16GB one you'd want for anything beyond DNS.

The Boards

Raspberry Pi 5 (8GB): Best Overall

Broadcom BCM2712, 4x Cortex-A76 @ 2.4GHz, LPDDR4X in 1/2/4/8/16GB variants, single 1GbE, and a PCIe 2.0 x1 interface that takes an M.2 HAT for NVMe boot. The db-tronic starter kit I've linked throughout this site* bundles the 8GB board with a case, fan, official power supply, and SD card, which saves a separate shopping list if you don't already have those parts lying around.

The Pi 5's real advantage over its predecessor and most alternatives isn't raw compute. It's the PCIe lane and the official HAT ecosystem around it, which finally make NVMe boot a well-documented, first-class option instead of a community hack. For homelab edge roles (DNS, Keepalived, Home Assistant, lightweight k3s agents) the Pi 5 has enough CPU headroom, and the software maturity means you spend your time on the service instead of debugging why a USB Wi-Fi dongle or camera module doesn't work on one particular board revision.

Pros: unmatched software/community support, official case and HAT ecosystem, production commitment to 2036, PoE+ via HAT
Cons: single 1GbE only, NVMe needs a separate HAT, price has crept up since the Pi 4 era

Radxa Rock 5B: Best Raw Performance

Rockchip RK3588 (4x Cortex-A76 + 4x Cortex-A55), 4/8/16GB LPDDR4X, one 2.5GbE port, and an on-board M.2 M-key slot (PCIe 3.0 x4) for NVMe, so no HAT needed.

On paper this is a much more powerful board than the Pi 5. Eight cores, the RK3588's hardware video codecs, and its 6 TOPS NPU give it an edge for anything that can use that hardware (video transcoding, some ML inference), and the on-board M.2 slot is a real convenience. The tradeoff is software maturity: mainline kernel support has improved a lot, but you're more likely to hit a driver gap or need a vendor image for full hardware support. NPU acceleration in particular is still the least mature part of the RK3588 software stack across every board, not just this one.

Pros: more compute headroom than the Pi 5, on-board NVMe slot with four PCIe 3.0 lanes, 2.5GbE
Cons: smaller community than Pi, less mature driver/firmware stack, shorter and less predictable support window, fewer case/accessory options

Orange Pi 5 Plus: Best Specs-per-Euro

Same RK3588 SoC as the Rock 5B, 4/8/16/32GB RAM, dual 2.5GbE, on-board M.2 M-key slot (PCIe 3.0 x4).

Orange Pi has historically undercut Radxa on price for the same silicon, and the 32GB configuration is unusual in this class. It's useful if the plan is running several lightweight VMs via KVM rather than bare containers. Orange Pi's documentation and image quality have historically been rougher than Radxa's, which are in turn rougher than Raspberry Pi's. If you hit a hardware quirk, expect more time in forum threads and less in polished documentation.

Pros: most RAM headroom in this comparison, dual 2.5GbE, on-board NVMe
Cons: documentation and image quality behind Radxa and Pi, smallest community of the three, needs an active cooler under sustained load

The Honest Alternative: Skip the SBC Entirely

For a homelab specifically (as opposed to embedded or GPIO-driven projects, where SBCs have no real substitute), the comparison that matters most often isn't Pi vs. RK3588 board. It's SBC vs. a budget x86 mini PC. A Beelink S12 Pro* with an Intel N100, 16GB DDR4, and a 500GB SSD runs full x86 Linux with zero driver-maturity questions. It costs more than a bare board, so compare it against the full price of an RK3588 board plus case, power supply, and NVMe drive. I cover it in more depth in the full mini PC comparison. If the workload doesn't need GPIO, camera interfaces, or the smallest possible footprint, the N100 box removes the whole "will this peripheral work on this SBC" category of problem.

Quick Comparison

	Pi 5 (8GB)	Rock 5B (16GB)	Orange Pi 5 Plus (16GB)
CPU	4x A76	4x A76 + 4x A55	4x A76 + 4x A55
NPU	None (optional AI HAT+ via PCIe)	6 TOPS	6 TOPS
Networking	1x GbE	1x 2.5GbE	2x 2.5GbE
Storage	microSD / NVMe via HAT (PCIe 2.0 x1)	microSD / on-board M.2 (PCIe 3.0 x4)	microSD / on-board M.2 (PCIe 3.0 x4)
Software maturity	Excellent	Good, improving	Fair, improving
Community/support	Largest by far	Moderate	Smaller

Who Should Buy Which

Buy the Raspberry Pi 5 if: you want the widest hardware/software compatibility, you're running GPIO or camera peripherals, or the workload is light (DNS, Keepalived, Home Assistant) and driver headaches aren't worth risking for compute you don't need.

Buy the Rock 5B or Orange Pi 5 Plus if: the workload is CPU- or RAM-bound (multiple KVM VMs, video transcoding, light ML inference) and you're comfortable working through a smaller community's documentation when something doesn't work out of the box.

Buy an N100 mini PC instead if: the project doesn't need GPIO or the smallest physical footprint. For pure homelab service hosting, x86 driver maturity beats every SBC option here.

Frequently Asked Questions

Is the Rock 5B actually faster than the Raspberry Pi 5?

In multi-threaded benchmarks and anything that can use the RK3588's NPU or video codecs, yes, by a clear margin. In homelab service hosting (reverse proxies, DNS, lightweight containers) the difference is far less noticeable than the spec sheet suggests, because those workloads rarely saturate either board's CPU.

Should I buy an RK3588 board for k3s?

For a worker node running a handful of lightweight pods, either family works. The Pi 5 now goes up to 16GB, which covers most edge-node use. If you need more than that, the 32GB Orange Pi 5 Plus is the only option here. Budget extra time for kernel/driver troubleshooting on either RK3588 board if you hit an edge case.

Does the Raspberry Pi 5 support NVMe boot?

Yes, via the official M.2 HAT+ (or third-party equivalents) on the PCIe 2.0 x1 interface. It's a separate purchase, unlike the Rock 5B and Orange Pi 5 Plus, which have the M.2 slot on-board.

What about power draw?

All three boards idle in the low single-digit watts in published measurements. The differences are a few watts at most, not a category difference, and not enough on their own to decide anything for a homelab running a handful of these devices.

Conclusion

The Raspberry Pi 5 is no longer the best SBC on paper. The RK3588 boards from Radxa and Orange Pi out-spec it for similar money. What the Pi still wins decisively is the thing that costs you time in a homelab: software and driver maturity. For DNS, Keepalived, and similar edge roles, I'm staying on the Raspberry Pi platform, and the Pi 5 is what I'd buy to replace my 4Bs. For CPU- or RAM-bound work where you'll trade some driver risk for more silicon, the Rock 5B or Orange Pi 5 Plus is worth it. And for pure service hosting with no GPIO requirement, a budget x86 mini PC often beats every SBC in this comparison on total reliability per euro.

Unable to mkstemp /tmp/apt.sig: How /tmp 0755 Broke apt

david — Mon, 09 Nov 2026 15:42:24 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

For about two months, apt on my Proxmox host could not refresh a single repository. /tmp had drifted from mode 1777 to 0755, apt's unprivileged _apt sandbox user could no longer create its signature temp files, and every apt-get update quietly fell back to stale index files. The fix was one chmod, plus a watchdog check so the next drift raises an alert instead of waiting for me to trip over it.

A week later I found a second version of the same problem on the Raspberry Pi nodes: unattended-upgrades ran every day, logged success, and left 69 updates untouched, including Docker Engine and containerd. Both failures share a shape. The update machinery was running, and nothing was getting updated.

View the complete homelab infrastructure source on GitHub 🐙

How it surfaced: not through an alert

No alert caught the /tmp problem. I was rolling out a change to the host watchdog on pve-mgmt-01, the single Proxmox box everything else in the homelab runs on, and the Ansible playbook died on its very first task:

Failed to update apt cache: unknown reason

"Unknown reason" is the Ansible apt module passing on that apt did not exit cleanly, without saying why. Running apt-get update by hand on the host showed the real error:

Unable to mkstemp /tmp/apt.sig.<XXX> - GetTempFile (13: Permission denied)

followed by the warning that matters most and gets read least:

The repository is not updated and the previous index files will be used

That warning came up for every repository. apt-get update still completed and printed only W: lines, so a quick glance looked like a noisy but successful run. In reality the host had not accepted a fresh package index since the problem began, and any update or install went against outdated metadata.

Root cause: /tmp at 0755, and who apt runs as

stat on /tmp showed the problem immediately:

$ stat -c %A /tmp
drwxr-xr-x

That is root:root 0755. A normal /tmp is drwxrwxrwt, mode 1777: world-writable, with the sticky bit so users can only delete their own files.

The reason this only broke apt is privilege separation. apt does not do its downloading and signature checking as root. It drops to a sandbox user, _apt, configured by APT::Sandbox::User, and that user writes temporary signature files to /tmp while verifying repository metadata. With /tmp at 0755, only root can create files there. _apt gets Permission denied, signature verification fails for every source, and apt keeps the old indexes.

Root, meanwhile, can still write to /tmp without complaint. Every root-run cron job, every script, every Proxmox service I touched kept working. Nothing else on the host needed a non-root user to write a temp file, so nothing else complained. That is why this went unnoticed for two months.

How long, and why I can't say what did it

/tmp on this host is a tmpfs from the stock systemd unit tmp.mount, with these options:

Options=mode=1777,strictatime,nosuid,nodev,size=50%,nr_inodes=1m

The running mount matched the unit on size and nr_inodes, but mode and strictatime were missing from its effective options. The mount had been active since 2026-08-23 12:29:28 CEST, which puts the broken state at roughly two months by the time I found it on October 16.

I could not reconstruct the trigger. Nothing in the repository and nothing on the host (scripts, cron, shell history) runs chmod on /tmp. The most plausible explanation is a single manual chmod 0755, or a remount issued without mode=. I'd rather write "unknown" in the incident record than invent a cause that fits.

The fix

The live fix was one command:

chmod 1777 /tmp

After that, stat -c %A /tmp showed drwxrwxrwt again, and apt-get update fetched 898 kB with no signature errors.

Persistence needed no change. tmp.mount sets mode=1777 on every boot, so a reboot would have repaired it as well. The drift was a runtime state, not a broken boot configuration. That is also the uncomfortable part: if I don't know what changed the mode, I can't stop it from happening again. I can only make sure I hear about it.

Detection: one more check in the host watchdog

The Proxmox host already runs a watchdog script every five minutes that checks thin-pool usage, CPU temperature, load, stale snapshots and NVMe kernel errors, and sends everything it finds through ntfy with escalating backoff. Adding /tmp there costs seven lines in ansible/roles/pve_power/templates/pve-watchdog.sh.j2:

# /tmp must be world-writable + sticky (1777). A chmod to 0755 broke apt's
# _apt sandbox user for ~2 months (mkstemp: permission denied) with no other
# symptom; tmp.mount owns the mode at boot, this catches runtime drift.
TMP_MODE=$(stat -c %a /tmp 2>/dev/null)
[ "$TMP_MODE" != "1777" ] && \
  ALERT="${ALERT}/tmp mode is ${TMP_MODE} (want 1777). "

It appends to the same ALERT string as every other check, so it inherits the existing ntfy topic, the maintenance-mode suppression and the backoff without any new plumbing. The check detects a repeat. It does not prevent one, and the incident record says so in those words.

A week later: unattended-upgrades running, nothing upgrading

On October 22 the pending-update counts across the hosts told two different stories. The Debian hosts looked fine: 0 to 6 pending upgrades on the k3s nodes, the NFS server and Atlantis. The two Raspberry Pi 4B nodes that run AdGuard and Unbound behind keepalived had 69 pending updates, and that number wasn't moving.

unattended-upgrades was installed and enabled there too, and its log showed it running daily. The log line that explained everything was the one listing what it was allowed to touch:

INFO Allowed origins are: origin=Debian,codename=trixie,label=Debian,
     origin=Debian,codename=trixie,label=Debian-Security,
     origin=Debian,codename=trixie-security,label=Debian-Security

Only Debian. The Pi nodes also carry repositories from the Raspberry Pi Foundation, Docker CE, Tailscale and AZlux, and packages from those were never candidates. The backlog included docker-ce (29.5.0 installed, 29.9.2 available), containerd.io (2.2.3 installed, 2.4.2 available), libc6, openssl and the Raspberry Pi firmware packages. The installed containerd was affected by five published CVEs, the worst rated 9.9 critical, all fixed in the 2.4.2 that sat in the repository.

Matching third-party origins is fiddly

The fix lives in the common Ansible role, which runs on every managed host:

- name: Let unattended-upgrades cover the third-party repositories
  ansible.builtin.blockinfile:
    path: /etc/apt/apt.conf.d/50unattended-upgrades
    marker: "# {mark} ANSIBLE MANAGED ORIGINS"
    block: |
      Unattended-Upgrade::Origins-Pattern {
          "origin=Debian,codename=${distro_codename},label=Debian";
          "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
          "origin=Debian,codename=${distro_codename}-updates,label=Debian";
          "origin=Raspberry Pi Foundation,codename=${distro_codename},label=Raspberry Pi Foundation,c=main";
          "origin=Docker,label=Docker CE,c=stable";
          "origin=Tailscale,codename=bookworm,label=Tailscale,c=main";
      };

Each third-party line uses different fields because the repositories publish different metadata. Docker CE has no codename at all, so a codename= match finds nothing. Tailscale has no suite but does publish codename bookworm, even on these trixie hosts. My first two attempts failed on exactly this: matching Docker on codename= matched nothing, and matching it on n= matched nothing either, because n= is the codename field, not the suite.

What worked was not guessing. apt-get indextargets prints origin, label, suite, codename and component for every index the host knows, so the right field combination for each repository can be read off directly. unattended-upgrade --dry-run --debug then shows the allowed origins and the package list they produce, which confirms the pattern before it runs for real.

Applied live, unattended-upgrades installed 74 packages on both Pi nodes: Docker 29.5.0 to 29.9.2, containerd 2.2.3 to 2.4.2, Tailscale 1.98.4 to 1.104.2, kernel 6.18.29 to 6.18.50, and all containers came back up.

The trade-offs I accepted

Letting unattended-upgrades touch Docker has a cost. A Docker Engine or containerd update restarts the daemon, which stops every container on that node, including AdGuard and Unbound. The keepalived VIP moves to the other Pi, so DNS stays up, but the containers on the updating node still restart. I accept that over a container runtime that sits behind on published security fixes.

Kernel and firmware updates still need a reboot. unattended-upgrades installs them and writes /var/run/reboot-required, but Automatic-Reboot stays off. An unattended reboot of the DNS and proxy nodes is not something I want, so reboots remain a deliberate, announced step. That fits the split I described in what auto-updates and what needs a human.

The k3s cluster is not covered by any of this. k3s ships its own containerd inside the binary, so the packaged containerd.io version says nothing about it. Its containerd is fixed only by a k3s upgrade, which is a separate job with a larger blast radius than a package update.

What I check now

apt warnings are failures. The repository is not updated and the previous index files will be used means the update did not happen, whatever the exit status says. A W: line from apt-get update deserves the same attention as an E: line.
Test as the user that does the work. Root writing to /tmp proved nothing. The failing write came from _apt. When something only breaks for an unprivileged service user, a root shell will tell you everything is fine.
"unattended-upgrades ran" is not "the host is patched". The log line to read is Allowed origins are:, and the number to watch is pending updates per host over time. A count that doesn't go down is the alert.
Detection when the cause is unknown. I couldn't find what changed /tmp, so the honest fix is a check that pages me within five minutes of a repeat, not a guess at a root cause.

This is the same lesson as the Pi edge hardening work: an unattended box needs a check for each way it can fail quietly, because nobody is watching its console.

If file modes, the sticky bit and what stat -c %a actually prints are still a bit hazy, The Linux Command Line* covers permissions and special modes more clearly than most man pages.

Prometheus Sizing for Homelab: RAM, Retention, and a 2Gi Limit

david — Mon, 09 Nov 2026 15:41:48 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). As an Amazon Associate I earn from qualifying purchases. Buying through them costs you nothing extra.

My Prometheus ran for months with no resource requests or limits. Nobody noticed until it landed on vm-srv-k3s-13, at the time the smallest node in the cluster with 4GB, and used about 1.4Gi there. Together with everything else scheduled on that node it pushed memory to 100%, and containerd kill timeouts started restarting unrelated pods. Three weeks later the TSDB was filling its NFS volume. This post covers the values that fixed both, what the live numbers look like now, and one restart problem that looked like memory but wasn't. It's the same kube-prometheus-stack deployment behind the cluster's alerting, so when Prometheus falls over, the alert pipeline goes with it.

View the complete homelab infrastructure source on GitHub 🐙

What Drives Prometheus Memory

Prometheus keeps the most recent data (roughly the last two to three hours) in an in-memory head block and writes older data to disk as compacted blocks. Memory follows from that:

Active series count is the biggest factor. Every unique label combination is a series, and each one in the head block costs memory.
Scrape interval doesn't add series, but it adds samples, which means more head chunks per series and more data to compact.
Retention is mostly a disk cost. Older blocks are memory-mapped and paged in when queried, so long-range queries over long retention still cost RAM.
Recording rules produce new series of their own.

Target count is a poor predictor. My Prometheus scrapes 39 targets across 25 jobs and holds about 204,000 head series. Almost all of that comes from two jobs: the kubelet (cAdvisor container metrics) returns about 97,000 samples per scrape and the API server about 52,000. The largest single metrics are API server latency histograms like apiserver_request_duration_seconds_bucket. One detail I didn't expect: the API server still exports etcd_request_duration_seconds_bucket (about 22,000 series), even though this cluster's datastore is SQLite through kine, not etcd.

Scrape Interval: 60s Globally

The chart default is 30 seconds. I set it to 60:

prometheus:
  prometheusSpec:
    # Scrape every 60s instead of chart default 30s: halves scrape
    # load on the control-plane node and TSDB retention growth; 15s
    # granularity is far below any alert window we configure.
    scrapeInterval: 60s

That halves the samples written per series compared to the default. It applies to the chart's own ServiceMonitors (kubelet, API server, node-exporter, kube-state-metrics) unless a ServiceMonitor sets its own interval. The trade-off is that rate() windows need to be at least a few minutes wide to hold enough samples. With 60-second scrapes, a [1m] range often contains a single sample and returns nothing. My alerts all look back several minutes or longer, so that cost is zero in practice.

Extra jobs go under prometheus.prometheusSpec.additionalScrapeConfigs, not at the top level of the values file. They inherit the global interval unless they set scrape_interval themselves. Mine cover node-exporters outside the cluster (the Raspberry Pis, the Docker LXC, the Proxmox host), the Proxmox and SNMP exporters, and a few blackbox probes:

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
      - job_name: 'node-exporter-rpi'
        static_configs:
          - targets:
              - '10.0.20.2:9100'
              - '10.0.20.3:9100'
            labels:
              group: 'rpi_nodes'
        relabel_configs:
          - source_labels: [__address__]
            target_label: instance

These are small. The out-of-cluster node-exporters each return a few thousand samples per scrape, an order of magnitude below the kubelet.

Retention: 30 Days, Capped at 12GiB

The TSDB lives on the nfs-client storage class, not on the Proxmox host's local NVMe. It ran with retention: 30d and nothing else until it started filling the NFS volume. The fix was a size cap:

prometheus:
  prometheusSpec:
    retention: 30d
    retentionSize: 12GiB
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: nfs-client
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 20Gi

retentionSize sits on top of the time-based retention. Persisted blocks, the WAL, and memory-mapped head chunks all count toward it, and when the total goes over, Prometheus deletes the oldest blocks regardless of retention. Deletion happens block by block, so usage can sit a bit above the cap. Leave headroom on the volume. Also note that nfs-subdir-external-provisioner does not enforce the PVC size; the 20Gi request is a label, and the real limit is the free space on the NFS export.

In practice the size cap is now the limit that matters. Over the last 30 days, size-based retention deleted blocks nine times and time-based retention deleted none. The oldest sample in the TSDB is 21 days old, not 30. Blocks take about 10.9GB and the WAL about 0.2GB. If I wanted the full 30 days back, the lever would be fewer series (dropping unused kubelet and API server histogram buckets with metricRelabelings), not a bigger cap.

WAL compression is not set in my values. It has been on by default since Prometheus 2.20, and this cluster runs Prometheus 3.12. If you do set it explicitly, walCompression in the Prometheus Operator CRD is a boolean (walCompression: true), not a string like enabled.

Resources and Placement

After the k3s-13 incident, Prometheus got explicit requests and limits so the scheduler accounts for its real footprint, plus a soft preference away from both the control-plane node and the smallest worker:

prometheus:
  prometheusSpec:
    affinity:
      nodeAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
                - key: kubernetes.io/hostname
                  operator: NotIn
                  values:
                    - vm-srv-k3s-11
                    - vm-srv-k3s-13
    resources:
      requests:
        cpu: 100m
        memory: 1Gi
      limits:
        cpu: 1000m
        memory: 2Gi

The current numbers, read from Prometheus itself on 2026-11-04:

Metric	Value
Scrape targets	39 (25 jobs)
Head series	~204,000
Samples appended	~3,800/s
Resident memory (process)	~1.1GB
Container working set	~1.7GB now, ~1.9GB peak over 14 days
Memory limit	2Gi
TSDB on disk	~10.9GB blocks + ~0.2GB WAL
Oldest data	21 days

The working set includes page cache from memory-mapped blocks, which the kernel can reclaim, so it overstates what Prometheus strictly needs. Still, a 14-day peak at roughly 88% of the limit is not much headroom. If series count grows, the 2Gi limit is the next thing to revisit, together with trimming the kubelet and API server histograms.

The Restarts That Weren't Memory

In early October Prometheus had restarted 46 times, and with a 2Gi limit fresh in place, OOM kills were the obvious suspect. None of them were. The container's lastState showed reason=Completed, exit code 0: kubelet had sent SIGTERM after a failed liveness probe, and Prometheus shut down cleanly. The chart-default probe allows 3 seconds to answer /-/healthy and restarts after about 30 seconds of failures. During TSDB compaction stalls, and while the Proxmox host (a single BMAX Mini PC*) sat at a load of 38 to 56 during a backup window, Prometheus couldn't answer in time.

The fix was a more tolerant probe. The first attempt set prometheusSpec.livenessProbe, which looks right and does nothing: the Prometheus CRD has no such field, structural schema pruning dropped it on apply, and ArgoCD still reported Synced. Probe overrides have to go into the container override list:

prometheus:
  prometheusSpec:
    containers:
      - name: prometheus
        livenessProbe:
          periodSeconds: 15
          timeoutSeconds: 10
          failureThreshold: 12
        readinessProbe:
          periodSeconds: 15
          timeoutSeconds: 10
          failureThreshold: 4

That gives three minutes of tolerance before a restart. The Prometheus container has restarted once since. Before raising a memory limit because of restarts, check the exit code. OOMKilled with exit 137 is memory; Completed with exit 0 is a probe.

Recording Rules

The chart ships 48 rule groups with 259 rules in my setup, recording and alerting combined. Each recording rule produces series of its own, evaluated continuously whether or not anything reads them. I haven't disabled any rule groups wholesale. What I did turn off are the chart components that can never work on k3s: kubeControllerManager, kubeScheduler, kubeProxy, and kubeEtcd. k3s runs those inside the single server binary and doesn't expose them as scrapable endpoints, and this cluster has no etcd at all. Disabling them removed their ServiceMonitors and the alerts that fired permanently because of them.

If you want to trim further, the defaultRules.rules map in the chart toggles individual groups. Key names change between chart versions, so read your chart's values.yaml first. Be careful: the bundled Grafana dashboards query many of the recorded series, and several of those groups are alerting rules, not recording rules. Turning one off removes alerts, not just series.

Takeaway

Set requests and limits on Prometheus from day one, and pick its node on purpose; mine took down a 4GB worker before anyone gave it a budget. Cap the TSDB with retentionSize, because on a small NFS volume the size limit will win over the time limit. And when Prometheus restarts, look at the exit code before blaming memory. None of this is visible one layer down: the two-tier overcommit guard checks Proxmox memory allocations per VM, and it can't see one pod eating most of a VM's RAM. Inside the cluster, requests and limits are that guard.

CloudNativePG on k3s: PostgreSQL That Survives Node Failures

david — Mon, 02 Nov 2026 15:18:25 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Nextcloud's PostgreSQL database ran as a StatefulSet with a Longhorn PVC. When the Proxmox host rebooted for a kernel update, Longhorn volumes took 90 seconds to reconstruct. During that window, Nextcloud returned 500 errors — the same Longhorn Multi-Attach problem that pushed everything else off distributed storage on this single-host cluster. The fix: CloudNativePG (CNPG), a Kubernetes operator that manages PostgreSQL streaming replication, automatic failover, and backups — without depending on a distributed storage layer.

View the complete homelab infrastructure source on GitHub 🐙

Why Not Just StatefulSets?

A StatefulSet with a PersistentVolumeClaim works until the node dies. In a single-node homelab (one BMAX Mini PC* running Proxmox with k3s VMs), "node dies" means "Proxmox host reboots." Longhorn reconstructs the volume from replicas, but the reconstruction takes time. During that time, the PostgreSQL pod is in ContainerCreating or CrashLoopBackOff.

CNPG solves this differently: it manages PostgreSQL replication at the database level, not the storage level. A primary and a replica run on different nodes. If the primary's node dies, CNPG promotes the replica to primary. No storage reconstruction needed.

Installing CNPG

The CNPG operator is deployed with a Helm chart:

helm repo add cnpg https://cloudnative-pg.github.io/charts
helm install cnpg cnpg/cloudnative-pg \
  --namespace cnpg-system \
  --create-namespace

The operator runs in the cnpg-system namespace and watches for Cluster CRDs.

The Cluster Definition

A CNPG PostgreSQL cluster with streaming replication:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: nextcloud-db
  namespace: nextcloud
spec:
  instances: 2
  postgresql:
    parameters:
      max_connections: "200"
      shared_buffers: "256MB"
    pg_hba:
      - host all all 10.0.0.0/8 md5
  storage:
    size: 10Gi
    storageClass: local-path
  backup:
    barmanObjectStore:
      destinationPath: "s3://backups/nextcloud-db"
      endpointURL: "http://minio.minio.svc:9000"
      s3Credentials:
        accessKeyId:
          name: minio-credentials
          key: ACCESS_KEY_ID
        secretAccessKey:
          name: minio-credentials
          key: SECRET_ACCESS_KEY
    retentionPolicy: "30d"
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "1"

instances: 2 means one primary and one replica. CNPG handles streaming replication automatically. The pg_hba entry allows connections from the 10.0.0.0/8 subnet (the homelab VLAN range).

The Backup Sidecar

CNPG uses Barman for backups. The barmanObjectStore section points to a MinIO instance running in the cluster, kept deliberately separate from the Garage bucket Velero uses. Backups run daily by default. The retentionPolicy: "30d" keeps 30 days of backups.

To trigger a manual backup:

kubectl cnpg backup nextcloud-db -n nextcloud

To restore:

kubectl cnpg restore nextcloud-db-backup-20261101 -n nextcloud

Failover in Practice

When the primary node fails:

CNPG detects the primary is unreachable (timeout: 30 seconds)
CNPG promotes the replica to primary
CNPG creates a new pod for the old primary (as a replica)
The new primary accepts writes

The failover takes 30-60 seconds. During that window, PostgreSQL connections are dropped. Application-level retries (Nextcloud has them) handle the gap transparently.

The CNPG operator logs the failover event:

promoted instance nextcloud-db-1 to primary
created new replica pod nextcloud-db-0

Monitoring CNPG

CNPG exposes Prometheus metrics through a ServiceMonitor:

cnpg_pg_stat_activity_count{datname="nextcloud",state="active"} 12
cnpg_replication_lag_seconds{instance="nextcloud-db-1"} 0.003
cnpg_backups_total{status="ok"} 30

Grafana dashboard: alert if replication_lag_seconds exceeds 5 seconds. That means the replica is falling behind, and a failover would lose data.

The Trade-off

CNPG adds complexity: an operator, Barman for backups, and streaming replication configuration. The benefit is PostgreSQL that survives node failures without depending on Longhorn's storage reconstruction. For a single-node homelab, this is the right trade-off — the BMAX Mini PC* has enough RAM for two PostgreSQL instances, and the failover is faster than waiting for Longhorn to rebuild.

Traefik Rate Limiting That Actually Works on k3s

david — Mon, 02 Nov 2026 15:12:39 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Prometheus scraped Grafana 30 times in 10 seconds. Grafana's API returned 429s. The dashboard went blank. The root cause: a misconfigured Prometheus service monitor with a 1-second scrape interval — the kind of runaway scrape config that also drives Prometheus's own RAM usage. The fix: Traefik's RateLimit middleware, correctly configured to protect against scrape storms without blocking normal dashboard access. This runs on a BMAX Mini PC* with 20+ services behind Traefik — one misconfigured scrape can cascade into a cluster-wide outage.

View the complete homelab infrastructure source on GitHub 🐙

The Scrape Storm

Grafana runs on k3s behind Traefik. The Prometheus Operator's ServiceMonitor for Grafana had a scrapeInterval: 1s — a copy-paste error from a debug session. Prometheus fired 30 requests per second at Grafana's /api/datasources endpoint. Grafana's Go runtime couldn't handle the concurrent connections, started returning 503s, and the dashboard went blank.

Without rate limiting, any misconfigured scraper — or any user refreshing aggressively — can DoS a service behind Traefik.

The RateLimit Middleware

Traefik's RateLimit middleware uses a token bucket algorithm. It's configured per-entrypoint or per-route:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: rate-limit
  namespace: traefik
spec:
  rateLimit:
    average: 100
    burst: 200
    period: 1s
    sourceCriterion:
      ipStrategy:
        depth: 1

average: 100 — 100 requests per second per source IP
burst: 200 — allow bursts up to 200 requests
period: 1s — the averaging window
sourceCriterion.ipStrategy.depth: 1 — trust the first X-Forwarded-For header (correct behind one proxy)

The depth: 1 is critical. Without it, Traefik uses the raw TCP connection IP, which is always the Traefik pod's IP in a Kubernetes cluster. Every client gets the same rate limit bucket. With depth: 1, Traefik reads the real client IP from X-Forwarded-For.

Applying It Per Service

Not every service needs the same rate limit. Grafana's dashboard needs a higher burst than the API:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: grafana-rate-limit
  namespace: monitoring
spec:
  rateLimit:
    average: 50
    burst: 100
    period: 1s
    sourceCriterion:
      ipStrategy:
        depth: 1
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: grafana
  namespace: monitoring
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`grafana.lab.woitzik.dev`)
      kind: Rule
      middlewares:
        - name: grafana-rate-limit
          namespace: monitoring
      services:
        - name: grafana
          port: 3000
  tls:
    secretName: lab-wildcard-tls

Whitelisting Internal Traffic

Prometheus, ArgoCD, and other internal services shouldn't be rate-limited. Traefik's RateLimit middleware supports requestHost and requestHeaders as source criteria, but the cleanest approach is a separate middleware for internal services:

apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: internal-rate-limit
  namespace: traefik
spec:
  rateLimit:
    average: 500
    burst: 1000
    period: 1s
    sourceCriterion:
      requestHost: {}

Internal services use requestHost (the Host header) as the source instead of IP. Since internal services all share the same cluster IP, IP-based rate limiting would bucket them all together.

The 429 Response

When a rate limit is hit, Traefik returns a 429 Too Many Requests with a Retry-After header. The client should respect this. Most HTTP clients don't. The Grafana dashboard showed a blank page because the browser retried immediately. Adding a client-side backoff in the Prometheus scraper config fixed the retry behavior:

scrapeConfig:
  - scrapeInterval: 30s
    scrapeTimeout: 10s
    metricsPath: /metrics

The real fix was the scrape interval. The rate limit was the safety net.

Monitoring Rate Limits

Traefik exposes rate limit metrics in Prometheus format:

traefikmiddleware_ratelimit_requests_total{middleware="monitoring-grafana-rate-limit"} 42
traefikmiddleware_ratelimit_rejected_total{middleware="monitoring-grafana-rate-limit"} 3

Grafana alert: if rejected_total increases by more than 10 per minute, something is hammering the service. This is the early warning for scrape storms, bot attacks, or misconfigured clients.

Vault Configuration Under Terraform: Staged Migration Without Downtime

david — Mon, 26 Oct 2026 12:33:03 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Vault is the trust root for every other secret on the cluster. 25+ services depend on External Secrets Operator syncing from Vault's KV v2 engine — all running on a single BMAX Mini PC*. Authelia, databases, ArgoCD notifications, Velero credentials — all of them pull from secret/<app>/* paths. Unsealing this Vault after a restart is a separate problem, solved by the polling sidecar pattern — this article only covers what's configured once Vault is already up.

Vault's configuration — policies, Kubernetes auth method, roles — was managed through vault login and CLI commands. No version control, no audit trail, no reproducibility. If someone changed a policy by hand, nobody else knew.

The goal: bring Vault's configuration (not secret values) under Terraform. Staged, low-risk, each stage independently reversible.

View the complete homelab infrastructure source on GitHub 🐙

The Scope Boundary

The hardest part of Terraforming Vault is knowing what NOT to Terraform:

In scope (configuration)	Out of scope (stays manual)
Vault policies	Actual secret values (`secret/<app>/*`)
Auth methods (Kubernetes auth)	Unseal keys, root tokens
Secrets engine mount config	Vault server config (Helm, Raft)
Auth roles and bindings	TLS (currently disabled)

Secret values stay in Ansible Vault. Terraform manages configuration, never secret data. The Ansible Vault file (ansible/group_vars/all/vault.yml) is the durable source. ESO syncs from Vault to Kubernetes Secrets. If Terraform state is lost, you re-import config — no secrets leak.

This is the critical principle: Terraform state describes access rules, not the secrets themselves. Losing state means re-importing policies and roles, not exposing credentials.

The Six Stages

Stage 1: Policies (lowest risk)

Import existing Vault policies into Terraform. Policies are purely declarative — they describe access rules but don't change anything about Vault's operation.

resource "vault_policy" "eso" {
  name   = "eso"
  policy = file("${path.module}/policies/eso.hcl")
}

One .hcl file per policy under policies/. Extracted from the live Vault instance (vault policy read <name>), committed to git.

Risk: Zero. Policies are read-only from Vault's perspective. No restart, no unseal, no client disruption.

Verification: terraform plan shows only import blocks matching live state. vault policy list unchanged after apply.

Rollback: terraform state rm vault_policy.<name>. Vault policies are stateless — removing from TF doesn't delete them from Vault.

Stage 2: KV v2 Secrets Engine Mount

Import the secret/ KV v2 mount configuration. This configures the engine's existence and settings (version, max versions) but NOT the secret values.

resource "vault_mount" "secret" {
  path        = "secret"
  type        = "kv"
  options     = { version = "2" }
  description = "KV v2 secrets engine — application secrets"
}

Risk: Low. If the mount already exists with the same config, terraform plan shows no changes. If there's a config diff, review carefully before applying.

Verification: vault read sys/mounts/secret/ output matches the HCL config. vault kv list secret/ still works — no disruption.

Stage 3: Kubernetes Auth Method

Import the Kubernetes auth method — the mechanism by which Vault trusts the k8s API server to authenticate pods.

resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
  path = "kubernetes"
}

resource "vault_kubernetes_auth_backend_config" "config" {
  backend            = vault_auth_backend.kubernetes.path
  kubernetes_host    = "https://kubernetes.default.svc"
  kubernetes_ca_cert = file("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt")
}

Risk: Medium. Auth method misconfiguration breaks every ESO sync. Do not apply a diff here blindly — if terraform plan shows changes (not just imports), stop and investigate.

Verification: Trigger one ESO sync (kubectl annotate externalsecret -n <ns> <name> force-sync=$(date)). Check 3-5 apps that depend on Vault secrets — no restarts, no errors.

Stage 4: Kubernetes Auth Roles

Import the roles that map Kubernetes ServiceAccounts to Vault policies. This is the binding between "which pod" and "what it can access."

resource "vault_kubernetes_auth_backend_role" "eso" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "eso"
  bound_service_account_names      = ["external-secrets"]
  bound_service_account_namespaces = ["external-secrets"]
  token_policies                   = ["eso"]
  token_ttl                        = 3600
}

Risk: Medium for NEW roles (changes what pods can access). Low for importing existing ones. Least-privilege check: verify each role's bound_service_account_names + token_policies against the live config.

Stage 5: Additional Auth Methods (deferred)

Userpass, OIDC, or other auth methods beyond Kubernetes. Currently only Kubernetes auth is in use (all Vault access goes through ESO). If no other auth methods exist, skip entirely.

Stage 6: App Policies (pure additive)

Import the per-app policies — one per ESO-managed secret path. Each policy controls exactly which secret path an app's role can read.

Risk: Lowest. Policies are additive, don't affect existing access until a role references them.

Why Not Just `vault write` Everything?

The alternative is simpler: keep using CLI commands, document them in a runbook, hope nobody drifts.

The problem with that approach:

No audit trail. Who changed the authelia policy last week? vault audit logs secret access, not policy changes.
No reproducibility. Recreating a Vault configuration from documentation is error-prone. Terraform state is the single source of truth.
No drift detection. terraform plan shows you exactly what changed. Without it, drift accumulates silently until something breaks at 3 AM.
No rollback. With CLI, undo means "remember what it was before" and vault write it back. With Terraform, terraform apply reverts to the last known-good state.

The staged approach means you can stop at any stage. If Stage 3 (auth method) makes you nervous, don't apply it — Stages 1 and 2 are already valuable and carry zero risk.

The Maintenance Window

Each stage takes 5-20 minutes. All stages are independently reversible. Total: ~1 hour.

Stage	Time	Impact
1. Policies	15 min	Zero — read-only import
2. KV mount	5 min	Zero — config import
3. K8s auth	10 min	Zero if import-only
4. Auth roles	15 min	Zero if import-only
6. App policies	20 min	Zero — additive

If any stage shows unexpected drift: STOP, investigate, do not apply. The diff tells you exactly what's different. Fix in git or in Vault manually, re-plan.

What Stays Manual Longest

Item	Why	When to revisit
Secret values	Ansible Vault is the durable source	Never — this is the correct pattern
Vault server config	Managed by ArgoCD + Helm	When Vault moves to non-ArgoCD install
Unseal keys / root tokens	Security-sensitive, manual by design	Never
Vault upgrades	Renovate handles via PR	Ongoing

Bringing Vault config under Terraform isn't about Terraform being better than CLI — it's about configuration being version-controlled, auditable, and reproducible. The staged approach means zero risk at each step: if a stage makes you nervous, don't apply it. The previous stages are already valuable. The hardest part isn't the implementation. It's drawing the scope boundary — knowing that secrets stay in Ansible Vault and Terraform only touches the access rules around them.

Further reading: For understanding the security model behind Vault's policy system, Zero Trust Networks by Evan Gilman and Doug Barth* covers the "never trust, always verify" principles that Vault's Kubernetes auth method implements. The staged migration pattern — import first, modify later — is a core operations concept covered in The Phoenix Project by Gene Kim* under the context of change management and reducing deployment risk.

Velero Backs Up to the Cluster It's Backing Up: Breaking the Backup Circularity

david — Mon, 19 Oct 2026 12:38:54 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

The circularity: Velero backs up into Garage S3, which runs inside the Kubernetes cluster it's backing up. Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from.

This is the backup equivalent of storing your offsite backup key in the safe you're backing up. The offsite target exists in config but isn't actually reachable — and you don't realize the gap until you need it.

View the complete homelab infrastructure source on GitHub 🐙

The Two-Problem Stack

Two issues compound into structural fragility:

Problem 1: Backup Circularity

Velero's backup storage location points to Garage, an S3-compatible object store running as a StatefulSet inside the cluster. Every Velero backup — daily snapshots of Vault, Authelia, databases, ArgoCD configs — lands in a bucket that depends on the cluster surviving.

Cluster lost → Garage PVC lost → Velero backup bucket gone → nothing to restore from

PBS backs up the VM/LXC disks to a Seagate 2TB External HDD*, but PBS runs on the same physical host — a BMAX Mini PC*. Simultaneous loss of host + USB HDD means total unrecoverability at the VM layer. Velero was the last safety net — except it was circular.

Problem 2: Torn-State Hazard

Two PVCs with different storage backends, different failure modes:

PVC	Storage	Location	Failure mode
`garage-meta`	`local-path`	k3s node	Node-pinned, survives if node survives
`garage-data`	NFS (`nfs-client`)	NFS LXC	Survives if NFS LXC survives

On 2026-07-12, an unclean Garage shutdown (OOM kill) left db.sqlite corrupted — the same NFS/SQLite locking incompatibility that eventually forced every embedded database off NFS. The metadata — SQLite, bucket/key ownership, access grants — and the actual object data were on different volumes with different rollback fates. A partial restore or node loss would leave one without the other, with no clean way to reconcile them.

The Four Options

Option A: Consolidate meta + data on one volume

Move garage-meta from local-path to NFS (same backend as garage-data). Both PVCs on the same storage class → same snapshot/restore fate.

Verdict: Fixes torn-state but not circularity. NFS for SQLite is proven fragile — the 07-12 corruption happened on NFS-backed data. Moving data to local-path requires hardware you don't have yet.

Option B: External off-cluster target (Cloudflare R2)

Activate the existing r2-offsite backup storage location. Velero dual-targets: in-cluster Garage for speed + off-cluster Cloudflare R2 for bootstrap.

Cluster lost → Garage gone → Velero still has offsite copy on R2
Bootstrap: rebuild cluster → ArgoCD → Velero → restore from R2

Cost: ~$0.015/GB/month on R2 (egress free). At current backup size (~5-10Gi), negligible. Risk: Near-zero — additive only, no existing backup path changes.

Verdict: Directly solves circularity. The existing R2 config is ready — needs credentials and a "go".

Option C: PBS as the primary recovery layer

Lean into PBS as the from-scratch recovery path. PBS already backs up all VMs/LXCs (including the k3s VMs that carry Garage's data). Velero becomes a convenience layer for single-app restore, not the bootstrap path.

Verdict: Already the real bootstrap path (DR Tier 1-4). Velero is Tier 5, only if Garage survived. PBS is the actual safety net. The circularity is a Velero-level problem, not a PBS-level one.

Option D: Combine B + C (what I chose)

Activate R2 offsite for Velero (breaks circularity) AND acknowledge PBS as the primary VM-level recovery layer. Velero becomes the single-app/namespace convenience tool with a real offsite target.

Layer	What it covers	Recovery path
PBS	VM/LXC disks, full host recovery	Restore VMs from PBS → cluster up
Velero + R2	App-level config, secrets, databases	ArgoCD → Velero → restore from R2
Garage rebuild	If PVCs are lost entirely	Terraform re-creates buckets, ESO re-seeds keys

Verdict: Covers both problems. R2 breaks circularity (zero risk, additive). PBS handles VM-level recovery (already there). Garage rebuild from scratch is the fallback if PVCs are lost — better than a partial restore that could leave metadata/data inconsistent.

The Recovery Model

Three independent layers, each covering a different failure scenario:

Scenario                          → Recovery path
─────────────────────────────────────────────────────
Single app crash                  → Kubernetes restart (self-heals)
Single PVC lost                   → Velero restore from R2 offsite
Full cluster lost (host survives) → PBS restores k3s VMs → Velero from R2
Host + USB HDD lost               → Insurance (nothing saves this)

The key insight: no single layer has to be perfect. Each layer covers the failure modes the layer below misses. Garage's torn-state problem becomes irrelevant when the recovery procedure is "rebuild from scratch" rather than "restore from backup."

Implementation (Zero Downtime)

The R2 activation was purely additive — no existing backup path changed:

Create Cloudflare R2 bucket (free tier)
Populate r2-backuplocation.yml with real R2 credentials via External Secrets Operator
Apply the daily-offsite Velero Schedule (already committed)
Run one test backup, verify with velero backup describe
Confirm PodVolumeBackup entries present for PVCs

The in-cluster Garage target stays. Velero now writes to both: Garage for fast local restores, R2 for bootstrap-from-scratch. If Garage is gone, R2 has everything.

What I'd Change

Implement R2 sooner. The config existed for weeks before activation. The circularity was a known risk — the "I'll do it later" mindset is what creates disaster recovery gaps.
Add a Velero backup age alert. Alert when the most recent successful backup is older than 48 hours. If backups silently stop, you want to know before you need one.
Test the full recovery path. PBS restore of k3s VMs → cluster boot → ArgoCD reconciliation → Velero restore from R2. An untested recovery path is a theory, not a plan.

Backup circularity is the most common structural flaw in self-hosted Kubernetes: the tool that protects you depends on the thing you're protecting. The fix is always the same — add an independent offsite layer. Cloudflare R2 costs nothing at homelab scale, and the R2 egress-free model means you're not penalized for actually using the restore. The hardest part isn't the implementation. It's admitting your current backup doesn't survive the scenario you built it for.

Further reading: For understanding the full recovery architecture, Designing Data-Intensive Applications by Martin Kleppmann* covers replication, backup strategies, and the failure modes that make torn-state hazards so dangerous. The Velero circularity is a specific case of the general problem Kleppmann describes: systems that depend on their own recovery path.

The DNS Drift That Wasn't DHCP: How Proxmox Cloud-Init Silently Injects Search Domains

david — Fri, 16 Oct 2026 09:41:18 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Every DNS lookup from every pod in my k3s cluster was appending home.lan to the query. External lookups like registry-1.docker.io became registry-1.docker.io.home.lan, failed locally, recursed all the way out, and added ~341ms of latency to every image pull, every API call, every webhook — traffic that should have gone straight to Unbound for recursive resolution, not round-tripped through a phantom search domain. I assumed DHCP was the source, wrote a Terraform fix, and watched the plan show zero diff against live state.

The real source was Proxmox's cloud-init generator — three layers above the DHCP server.

The Symptom

Prometheus showed p99 DNS latency at 341ms for external queries. Local queries (services within the cluster) resolved in under 1ms. The gap was consistent across all three k3s nodes.

# PromQL showing the split
quantile_over_time(0.99, dns_request_duration_seconds{domain!~".*cluster.local"}[5m])
# → ~341ms

quantile_over_time(0.99, dns_request_duration_seconds{domain=~".*cluster.local"}[5m])
# → ~0.8ms

The kubelet's DNS policy appends the node's search domain to every pod's /etc/resolv.conf. If the node has search home.lan, every pod inherits it. With the default ndots:5, any name with fewer than 5 dots gets the search domain appended first — meaning almost everything.

Wrong Hypothesis: DHCP

My first guess was the RouterOS DHCP server. The network was configured via Terraform, but I'd never explicitly set the domain field:

# Before fix — no domain attribute at all
resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
}

Adding domain = "" should force Terraform to detect and correct any drift:

resource "routeros_ip_dhcp_server_network" "vlan_networks" {
  for_each   = local.homelab_vlans
  address    = "10.0.${each.value}.0/24"
  gateway    = "10.0.${each.value}.1"
  dns_server = "10.0.20.5"
  domain     = ""  # explicitly empty — should fix drift
}

The plan showed zero changes. Live state already matched domain = "". DHCP was never the source.

Live Verification

Confirmed via the RouterOS REST API:

# Check all DHCP server config paths
curl -s "https://router.lan/rest/ip/dhcp-server" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dhcp-server/network" | jq '.[].domain'
# → all null

curl -s "https://router.lan/rest/ip/dns" | jq '.domain'
# → null

The RouterOS DHCP server had no domain/search config anywhere. domain = "" matched live state because the live state genuinely was empty. DHCP was never handing out home.lan.

The Real Source: Proxmox Cloud-Init

On the k3s nodes themselves:

networkctl status --json | jq '.DNS'
# → {"SearchDomains": ["home.lan"], "ConfigSource": "static"}

ConfigSource: static — not DHCP. The search domain was baked into the node's netplan config, generated by Proxmox's cloud-init writer.

When you create a VM in Proxmox with Terraform and leave initialization.dns.domain unset, Proxmox falls back to the PVE node's own configured search domain:

# On the Proxmox host
pvesh get /nodes/pve-mgmt-01/dns
# → search: home.lan

Proxmox's cloud-init generator reads the PVE node's DNS config and writes it into the VM's /etc/netplan/50-cloud-init.yaml. Every k3s VM got home.lan as its search domain — not from DHCP, but from the hypervisor itself.

Why This Is a Trap

The trap is the layering:

Proxmox PVE node DNS config
    ↓ (cloud-init fallback)
VM netplan search domain
    ↓ (kubelet DNS policy)
Pod /etc/resolv.conf search domain
    ↓ (ndots:5)
Every DNS lookup appends .home.lan
    ↓
~341ms latency on every external query

And the debugging trap: terraform plan showed zero diff on the DHCP resource because DHCP wasn't the source. A Terraform-native investigation would have concluded "no drift detected" and moved on. The real fix required:

Clearing the PVE node's default: pvesh set /nodes/pve-mgmt-01/dns -search ""
Regenerating cloud-init on each k3s VM
Declaring domain = "" in the Proxmox VM's initialization.dns block

Steps 1 and 2 are live changes to etcd-member VMs — no Terraform resource models this. Step 3 declares intent for future VMs but has no effect on already-provisioned ones (the dns block is in lifecycle.ignore_changes).

The Terraform Fix (Future VMs)

# terraform/stacks/proxmox/vm.tf
initialization {
  dns {
    domain  = ""  # explicitly unset — prevents PVE node fallback
    servers = ["10.0.20.5"]
  }
}

This ensures new VMs don't inherit the PVE node's search domain. Existing VMs need the manual cloud-init regen.

Lessons

1. DNS config has multiple sources. DHCP is the obvious one, but cloud-init, node-level config, and kubelet policy all contribute. Check all layers.

2. terraform plan zero-diff doesn't mean no drift. It means the specific resource you checked matches state. The drift might be in a different layer entirely.

3. networkctl status --json is your friend. The ConfigSource field tells you whether the search domain came from DHCP, static config, or another source. If it says "static", stop looking at DHCP.

4. Proxmox cloud-init is a hidden config source. If you leave initialization.dns.domain unset, Proxmox fills it from the PVE node. Always declare it explicitly. It's the same category of hidden-layer surprise as Proxmox's onboot attribute silently not round-tripping through Terraform — the provider abstraction has gaps that only show up once you go looking for drift that isn't there.

Further reading: DNS debugging keeps coming back to the same fundamentals — search domains, ndots, resolution order. Netzwerke: Verstehen, Einrichten, Administrieren* covers those fundamentals in more depth than this Proxmox-specific drift case gets into.

DNS latency from inherited search domains is invisible in most monitoring setups. You'll see high p99 latency on external queries but no obvious cause. If your pods have ndots:5 (the default) and your nodes have a search domain you didn't explicitly set, you're paying the tax on every lookup.

k3s vs Docker Swarm vs Nomad in 2026

david — Fri, 16 Oct 2026 09:40:43 +0000

Choosing between k3s, Docker Swarm, and Nomad stopped being a features question some time ago. It is now a project-trajectory question, and the three projects are on genuinely different trajectories. k3s tracks upstream Kubernetes within weeks and ships three supported release lines at once. Docker Swarm still works but its vendor has redirected development toward Kubernetes, and the Docker Engine v29 API floor broke parts of it badly. Nomad is the technically lightest option and the only one of the three that is not open source at all.

TL;DR — The Picks

Orchestrator	Pick it if	Main risk in 2026	License
k3s	You want Kubernetes, or your tooling already assumes it	You inherit the Kubernetes upgrade treadmill	Apache-2.0
Docker Swarm	You already run it and need cluster mode with no extra daemon	Docker Engine v29 dropped client and plugin compatibility	Apache-2.0
Nomad	You want one scheduler for mixed container and non-container workloads	BUSL source-available, not open source	BUSL → MPL-2.0 after 4 years

If you are starting fresh in 2026, the honest default is k3s. Not because it is the most capable — Nomad is lighter and Docker Swarm is simpler to learn — but because it is the only one of the three with a release cadence you can plan against, and because the surrounding ecosystem has consolidated around Kubernetes.

The three facts that decide it

Before features: these are the three things that actually constrain the decision, and all three are documented rather than opinion.

1. Docker Engine v29 broke Swarm compatibility

Docker Engine v29 raised the minimum supported daemon API version to 1.44, dropping all compatibility with clients built against Docker Engine older than v25. There is no fallback and no negotiation: tooling compiled against an older API receives a hard rejection.

The damage is not evenly distributed, and this is the part that matters if you run Swarm. Legacy (V1) Docker volume plugins are rejected by the v29 daemon. Because Swarm requires cluster-scoped persistent storage, Swarm operators hit this disproportionately. The failure mode is nasty: Docker keeps the volume metadata but cannot reach the plugin needed to mount the volume, so your data appears to be gone while it is still intact on the storage target.

There is a second trap in the same release. Upgrading some cluster nodes to v29 while others stay older — a staged rollout, the obvious upgrade strategy — causes encrypted overlay traffic to silently stop between v29.0.0 and v29.2.0. Silent is the operative word. If you use encrypted overlay networks, you must move all swarm hosts to v29.1.2 or later in one step; rolling upgrades are not available to you.

2. Docker has redirected Swarm development toward Kubernetes

Docker's own retired-products page is unusually direct about SwarmKit, the project that powers Swarm mode:

That is the vendor stating the trajectory. Swarm is not deprecated in the sense of being scheduled for removal — docker swarm init still works and the mode is still documented as built into the Engine — but it is a maintenance posture, not an investment posture. Portainer, the most visible Swarm management platform, published an advisory on 2026-03-10 recommending caution with continuing to run Swarm and planning a gradual move to Kubernetes.

3. Nomad is not open source

This one gets repeated wrong. Nomad Community Edition is licensed under the Business Source License. The change license is MPL 2.0, but it applies four years after the version is published. So a Nomad release from today becomes MPL 2.0 in 2030 — meaning that today, in 2026, if you need a genuinely open orchestrator, Nomad is not one.

The licensing situation is now stable but newly complicated. Since April 2026 HashiCorp's self-managed products follow IBM's enterprise software lifecycle, following the IBM acquisition. Nomad's support windows:

Version	Base support ends	Extended support ends	Ongoing extended ends
1.8.x (LTS)	2026-04-30	2026-04-30	2026-04-30
1.10.x (LTS)	2027-04-30	2027-04-30	2027-04-30
1.11.x	2026-10-31	2026-10-31	2026-10-31
2.0.x	2028-04-30	2029-04-30	2032-04-30

Note that 1.8.x is already fully out of support as of this writing, and 1.11.x ends 2026-10-31 — within weeks. Nomad v1.10.x is the last LTS line. From the 2.0.0 release onward, Nomad also drops semantic versioning for IBM's Version-Modification-Fix model, so tooling that parses Nomad version strings needs updating.

One operational warning worth knowing before you plan a mixed estate: a Nomad Enterprise cluster cannot be downgraded, and servers running open-source Nomad will panic if joined to an Enterprise cluster, because the Raft entries for Enterprise features are not parseable by the open-source build. Test that boundary before you need it.

k3s — the default for a reason

k3s is a conformant Kubernetes distribution packaged as a single binary. As of October 2026 it maintains three release lines concurrently: v1.36.4+k3s1 is current, with v1.35.8 and v1.34.11 still receiving backports. Each build pins its whole dependency set — for v1.34.11+k3s1 that is Kubernetes v1.34.11, Kine v0.16.4, SQLite 3.53.4, etcd v3.6.14-k3s1, containerd v2.2.7-k3s1, runc v1.4.2, Flannel v0.28.4, metrics-server v0.9.0, Traefik v3.7.13, CoreDNS v1.14.7, helm-controller v0.16.27 and local-path-provisioner v0.0.37.

That table matters more than it looks. When something breaks at 02:00 you can read which exact version of which component is running instead of discovering it. My own homelab runs k3s across several nodes, and that pinned-component view is what makes problems like ArgoCD cache staleness on a shared resource or picking the wrong storage backend for cluster state diagnosable at all.

The cost of k3s is that you inherit Kubernetes' upgrade treadmill. Upgrading three majors is not a hobby, and every Helm chart in your stack is a chance for the chart to be behind the API version you just moved to. One concrete example from the 1.34 line: the bundled Traefik chart moved to v40.x, which renamed the ingress-nginx migration provider from kubernetesIngressNginx to kubernetesIngressNGINX. That is a breaking change inside a patch-level chart bump.

I have written about the parts that bite in practice — the Velero and Garage S3 backup chain and Unbound with AdGuard and keepalived are both k3s deployments that took real iteration to get right.

Docker Swarm — simpler, and now on borrowed time

Swarm's genuine advantage is that it is already there. No second daemon, no new control plane, no new YAML dialect. If your Docker workflow is a compose file and a handful of services, docker stack deploy is a very small step up, and the service model with rolling updates and health checks is more capable than plain Compose.

The case against it in 2026 is the two items above: the v29 compatibility break with its silent overlay-traffic failure mode, and a vendor that has said in writing that development has slowed. Neither makes Swarm unusable tomorrow. Both mean every future upgrade is a risk you cannot get support for.

If you are on Swarm, the sensible 2026 posture is: pin the Engine version deliberately, verify you are past v29.1.2, audit for V1 volume plugins before you upgrade rather than after, and start introducing Kubernetes for new workloads so you are not migrating everything at once.

Nomad — the best scheduler, the weakest license

Setting licensing aside, Nomad is genuinely well-designed for its niche. It schedules not just containers but binaries, VMs and batch jobs through one job specification, it is built around a single deliberately small binary rather than a multi-component control plane, and its UI is easier to reason about than anything in the Kubernetes ecosystem. For a mixed estate — a few containers alongside a scheduled data job, an ETL step, a Windows task — that unification is the whole point and Kubernetes does not do it without extra machinery.

Against that: BUSL rather than an OSI license, a support matrix reshuffled by the IBM acquisition, a semantic-versioning break at 2.0, and an ecosystem that assumes Kubernetes. If your team already lives in Kubernetes YAML, Nomad's job syntax is a second language to maintain for a saving you have to measure yourself.

Head-to-Head

Dimension	k3s	Docker Swarm	Nomad
Scheduling unit	Pod	Service (task)	Job (group, task, service)
Non-container workloads	No	No	Yes — binaries, VMs, batch
Control plane footprint	Single binary per node	Built into the Docker Engine, no extra daemon	Single binary
License	Apache-2.0	Apache-2.0	BUSL, becomes MPL-2.0 after 4 years
2026 release cadence	Three concurrent K8s lines	Slowed, vendor-prioritised elsewhere	Spring/fall features, monthly as needed
Ecosystem direction	Consolidating around it	Maintenance	Stable, narrower
Upgrading is	Hard (K8s treadmill)	Easy	Moderate (mind the VMF versioning break)

Which one should you run?

Run k3s if your workloads are containers and you want the Kubernetes ecosystem — Helm, ArgoCD, Prometheus operators, the CNCF ecosystem at large. This is the default in 2026 and the choice I would make for anything new.

Stay on Docker Swarm only if you already run it in production and the migration cost is real. Treat it as a managed liability: pin versions, avoid the v29 trap, and plan the exit. Do not adopt it new in 2026.

Run Nomad if scheduling heterogeneous workload types on one control plane is worth more to you than an OSI license and a Kubernetes-shaped ecosystem. Just make that trade knowingly — you are choosing source-available software with a four-year conversion clock, not open source.

If you genuinely cannot decide between two, run both on the same hardware for a month with one real workload. The theoretical comparison above is the least interesting part; how each behaves with your actual data on your actual disks is what decides it.

Frequently Asked Questions

Is Docker Swarm being deprecated?

Not formally in the sense of a scheduled removal — docker swarm init still works and Swarm mode is still part of Docker Engine. But Docker's own documentation states that SwarmKit development "has slowed in favor of Kubernetes-based solutions," so treat it as maintenance-only. The practical risk arrived with Docker Engine v29, which broke client and plugin compatibility for Swarm disproportionately.

Is Nomad open source?

No. Nomad Community Edition is licensed under the Business Source License. Versions become MPL 2.0 four years after publication, so a release from 2026 becomes genuinely open source in 2030. If you require an OSI-approved license today, that rules Nomad out. k3s and Docker Swarm are both Apache-2.0.

Is k3s just "Kubernetes but worse"?

No. It is a conformant Kubernetes distribution — same API server, same controllers, same ecosystem — delivered as one binary with sensible defaults. The compromise is operational, not functional: you give up control over the component versions to get a maintainable install, which is why the pinned component table is worth reading before you debug anything.

Can I run Docker Swarm and Kubernetes on the same host?

Technically, yes, but you should not. Both want to own the container runtime and the network stack, and the interaction between a Swarm overlay network and a CNI is a debugging experience you do not need to have. Run them on separate machines.

What is the safest way to upgrade a Swarm cluster to Docker v29?

Get every host to v29.1.2 or later in a single step. Staged rollouts break encrypted overlay traffic silently in v29.0.0 through v29.2.0. Before you start, inventory your volume plugins — any V1 plugin will be rejected by the v29 daemon, and the resulting "missing" data is usually still on the storage target. Export your data first.

Does Nomad support Kubernetes workloads?

Not natively. Nomad is not a Kubernetes scheduler and does not run Kubernetes manifests. There is no compatibility path if you need the Kubernetes API; pick k3s instead. Nomad's own strength — scheduling binaries and batch jobs alongside containers — is precisely what Kubernetes does not offer.

How a YAML List vs Map Difference Crash-Looped My Kyverno Fix

david — Mon, 12 Oct 2026 14:54:50 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Kyverno's cleanup-controller was restarting every few hours. The leader-election timeout was expiring under load — the same class of leader-election instability that CPU contention causes for etcd — the controller lost its lock, and the new leader took over, causing a brief gap in policy enforcement. The fix looked simple: increase leaderElectionRetryPeriod. The fix crash-looped the very pod it was meant to stabilize.

The Symptom

cleanup-controller-7f8b9c4d6-xk2p4   0/1     CrashLoopBackOff   3 (42s ago)

The pod started, immediately crashed, and Kubernetes restarted it. Three crashes in under two minutes. The Kyverno dashboard showed the cleanup controller as unavailable, and policy reports stopped updating.

The Investigation

The crash-loop meant the binary was rejecting its own flags. Checking the pod's logs:

Error: unknown flag: --0=--leaderElectionRetryPeriod=10s

The binary received --0=--leaderElectionRetryPeriod=10s as a single flag. That's not a valid argument — it's a rendered artifact of wrong YAML structure.

The Root Cause: List vs Map

The Helm chart's cleanupController.extraArgs expects a map, not a list:

# WRONG — list syntax, renders as --0=--leaderElectionRetryPeriod=10s
cleanupController:
  extraArgs:
    - "--leaderElectionRetryPeriod=10s"

# CORRECT — map syntax, renders as --leaderElectionRetryPeriod=10s
cleanupController:
  extraArgs:
    leaderElectionRetryPeriod: "10s"

Helm's range function iterates over maps as $key, $value pairs. When you pass a list, Helm treats each list item as a map entry with an integer index as the key. The rendered result:

# What the binary actually receives:
--0=--leaderElectionRetryPeriod=10s

# What it expected:
--leaderElectionRetryPeriod=10s

The binary sees --0 as a flag name, doesn't recognize it, and exits with a non-zero code. Kubernetes restarts the pod. Crash-loop.

Why This Is a Production Risk

This wasn't a staging environment. The Kyverno cleanup-controller handles:

Policy report cleanup — old reports pile up without it
Namespace selector caching — stale cache entries cause policy enforcement delays
Leader election — the restart cycle itself destabilizes the leader lock

The irony: the fix for leader-election instability caused a worse instability. The cleanup-controller restarted 3 times before I caught it — and during those restarts, policy reports accumulated without cleanup.

How to Catch This Before It Hits Production

1. Dry-run the Helm template:

helm template kyverno kyverno/kyverno -n kyverno \
  --set cleanupController.extraArgs.leaderElectionRetryPeriod=10s \
  | grep -A5 cleanup-controller

If the rendered YAML shows --0=... in the container args, the structure is wrong.

2. Check the chart's values.yaml:

helm show values kyverno/kyverno | grep -A10 extraArgs

The chart documents whether extraArgs is a map or list. Trust the chart, not your muscle memory from other Helm charts.

3. Use Kyverno's policy reports to validate:

After deploying, check the cleanup controller's pod status immediately:

kubectl get pods -n kyverno -l app.kubernetes.io/name=kyverno-cleanup-controller

If it shows CrashLoopBackOff within the first minute, the flags are wrong.

The Fix

# kubernetes/system/kyverno/application.yml
cleanupController:
  extraArgs:
    leaderElectionRetryPeriod: "10s"

One line change. The pod started cleanly, the crash-loop stopped, and policy reports resumed normal cleanup.

The Pattern

This is a common Helm gotcha across many charts, not just Kyverno:

Chart	Flag field	Type	Gotcha
Kyverno	`extraArgs`	map	List syntax renders `--0=` prefix
Cert-Manager	`extraArgs`	map	Same pattern
Prometheus	`extraArgs`	map	Same pattern
ArgoCD	`extraArgs`	map	Same pattern

The fix is always the same: if the chart expects a map, use map syntax. If you're unsure, helm template will show you exactly what the binary receives.

Further reading: For a deeper dive into distributed systems failure modes, Designing Data-Intensive Applications by Martin Kleppmann* explains why leader-election timeouts behave the way they do under I/O pressure.

Kyverno policies are only effective when the controller is running. A crash-looping cleanup controller means policy reports accumulate without cleanup, and namespace selector caches go stale. Pin your Helm values correctly, and dry-run before deploying to production.

Three Fixes Were Merged. None of Them Had Taken Effect.

david — Mon, 12 Oct 2026 14:54:14 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

GitOps makes a specific promise: the cluster matches git. When a change is merged, it deploys. When a resource is deleted from git, it's deleted from the cluster. Merge = deploy is the core guarantee the entire workflow rests on.

It's not always true. This is a story about finding three cases where it wasn't - two discovered during a drift audit, one that would have been inert even if ArgoCD had synced it correctly.

View the complete homelab infrastructure source on GitHub 🐙

How the Gap Opens: ArgoCD's App-of-Apps Bootstrap Problem

The cluster uses ArgoCD's app-of-apps pattern: a root Application watches a directory, creates child Applications for each manifest it finds, and those child Applications sync the actual workloads. The root Application uses a directory.include glob to decide which files to track.

The problem: that glob is set once, at bootstrap time, and files added to the directory later are only tracked if they match the existing glob. A file that doesn't match is invisible to ArgoCD - it won't create an Application for it, won't sync it, and won't alert about it. From ArgoCD's perspective, the file doesn't exist.

The monitoring stack had this problem. Two files - application.yml and manifests-application.yml - had been added to the kubernetes/system/monitoring/ directory but were missing from the parent Application's directory.include glob:

# monitoring-bootstrap/application.yml (the root)
spec:
  source:
    directory:
      include: "alertmanager-config.yml"  # ← only this file tracked
      # application.yml and manifests-application.yml: invisible

A fix for a broken configSecret reference had been merged to application.yml. It was valid YAML, it would have worked - but ArgoCD never picked it up. The live cluster was still running the broken version from initial bootstrap, indefinitely.

Finding the Gap: What the Drift Audit Actually Checks

The standard ArgoCD health check (kubectl get applications -n argocd) doesn't catch this. All Applications show Synced/Healthy. The bootstrapped-but-untracked files don't generate an Application at all, so there's nothing to show as out-of-sync.

The audit that found it was manual: compare the contents of each kubernetes/system/*/ directory against the tracked files in the corresponding bootstrap Application, and flag any file not in the glob:

# For each subdirectory, list tracked files vs actual files
for dir in kubernetes/system/*/; do
  echo "=== $dir ==="
  # What ArgoCD tracks (from the bootstrap Application's directory.include)
  # vs what's actually in the directory
  ls "$dir"*.yml 2>/dev/null
done

The audit found approximately 14 directories with the same untracked-after-bootstrap gap - no current drift in most of them, but nothing preventing drift from accumulating silently if files were added there in the future.

This is the same class of gap that a GitHub Actions Terraform-plan-to-Slack alert is designed to close on the Terraform side — a green pipeline or a Synced badge tells you the command ran, not that the running state matches intent.

The First Live Finding: Traefik allowCrossNamespace

Traefik's allowCrossNamespace fix had been merged to git on 2026-07-04. The change allowed IngressRoutes in one namespace to reference Services in another - without it, argo.woitzik.dev, monitoring.woitzik.dev, and status.woitzik.dev returned bare 404s.

The fix was in git. ArgoCD showed the Traefik Application as Synced. But the endpoints were still 404ing.

The cause: the Traefik Application's last successful sync was before the fix merged. ArgoCD's sync status reflects the last successful reconciliation, not a continuous comparison. If the sync was clean before the change and nothing triggered a re-sync after (no new commit to the watched path, no manual refresh), the Application can show Synced while being hours or days behind.

# Force ArgoCD to re-compare against current git HEAD
argocd app sync traefik --force
# or via annotation:
kubectl annotate application traefik -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite

After manual sync, all three endpoints came back to 302 within seconds of the rollout completing.

The Second Finding: Velero's Fix That Couldn't Have Worked

Velero had a merged fix for repository maintenance frequency - too-frequent Kopia maintenance jobs were running during backup windows and causing timeouts. The fix set a custom maintenance frequency via Helm values.

The fix was synced correctly. It also did nothing, for two independent reasons:

Reason 1: Wrong Helm values structure.

The extraArgs key was placed at the top level of the Helm values instead of nested under configuration:

# What was merged (broken)
velero:
  extraArgs:
    - "--default-repo-maintain-frequency=168h"

# What Helm actually needs
velero:
  configuration:
    extraArgs:
      - "--default-repo-maintain-frequency=168h"

Helm silently drops unknown top-level keys. No error, no warning - the extra args simply weren't passed to the binary. Velero started with its defaults.

Reason 2: Wrong flag name.

The flag --default-repo-maintenance-frequency doesn't exist in Velero. The real flag is --default-repo-maintain-frequency (no "ance"). Even if the values structure had been correct, the binary would have printed an unknown flag error and ignored it.

Two independently-wrong things, each of which would have prevented the fix from taking effect on its own. Caught during the drift audit by checking the running Velero container's actual args:

kubectl get pod -n velero -l app.kubernetes.io/name=velero -o jsonpath=\
  '{.items[0].spec.containers[0].args}' | tr ',' '\n'
# → no --default-repo-maintain-frequency flag present

The correct fix, both issues resolved:

velero:
  configuration:
    extraArgs:
      - "--default-repo-maintain-frequency=168h"

The Three-Part Pattern

All three failures share the same shape:

	Traefik allowCrossNamespace	Velero maintenance frequency	Monitoring bootstrap files
Git state	Correct	Correct	Correct
Live state	Unsynced	Synced but inert	Never tracked
ArgoCD showed	`Synced/Healthy`	`Synced/Healthy`	No Application (invisible)
Found by	Manual sync + endpoint check	Checking running container args	Directory vs. glob comparison

In each case, the developer closed the PR with the reasonable belief that the fix was live. In each case, it wasn't. The merge was the easy part; the verification that the change actually took effect in the running cluster is the part that was skipped.

The Verification Step That Prevents It

For any change where "merged" isn't sufficient proof of "deployed," there's one check that works:

# For a Helm values change: verify the flag is in the running container
kubectl get pod -n <namespace> -l <label-selector> \
  -o jsonpath='{.items[0].spec.containers[0].args}'

# For a Kubernetes manifest change: verify the field is live
kubectl get <resource> -n <namespace> <name> \
  -o jsonpath='{.spec.<changed-field>}'

# For an ArgoCD Application bootstrap: verify the file is tracked
argocd app get <bootstrap-app> --output json | \
  jq '.spec.source.directory.include'

None of these are expensive. They're all one-liners. The cost of skipping them - in this case, Traefik returning 404 for three production subdomains, Velero running backup maintenance at the wrong frequency indefinitely, and monitoring fixes sitting inert - was substantially higher.

The same GitOps verification gap exists in Azure DevOps pipelines and GitHub Actions that deploy Terraform or Helm. A pipeline that reports "succeeded" proves that the deployment command ran without error - it doesn't prove that the cluster's running state matches the manifest. Checking the running state directly, after deployment, is the only check that closes the gap.

The Phoenix Project* is still the sharpest account I know of exactly this gap between "the change was made" and "the change took effect" - fiction, but painfully accurate about how it actually happens.

DEV Community: david

Gitea Actions on k3s: Self-Hosted CI Without GitHub

Gitea Actions: The GitHub Actions Clone

Deploying Gitea on k3s

The Action Runner

Workflow Migration

Docker-in-Docker

Resource Limits

The Trade-off

Raspberry Pi 5 Alternatives Compared 2026

TL;DR: The Picks

Why This Comparison Exists at All

Selection Criteria for Homelab Edge Use

The Boards

Raspberry Pi 5 (8GB): Best Overall

Radxa Rock 5B: Best Raw Performance

Orange Pi 5 Plus: Best Specs-per-Euro

The Honest Alternative: Skip the SBC Entirely

Quick Comparison

Who Should Buy Which

Frequently Asked Questions

Is the Rock 5B actually faster than the Raspberry Pi 5?

Should I buy an RK3588 board for k3s?

Does the Raspberry Pi 5 support NVMe boot?

What about power draw?

Conclusion

Unable to mkstemp /tmp/apt.sig: How /tmp 0755 Broke apt

How it surfaced: not through an alert

Root cause: /tmp at 0755, and who apt runs as

How long, and why I can't say what did it

The fix

Detection: one more check in the host watchdog

A week later: unattended-upgrades running, nothing upgrading

Matching third-party origins is fiddly

The trade-offs I accepted

What I check now

Prometheus Sizing for Homelab: RAM, Retention, and a 2Gi Limit

What Drives Prometheus Memory

Scrape Interval: 60s Globally

Retention: 30 Days, Capped at 12GiB

Resources and Placement

The Restarts That Weren't Memory

Recording Rules

Takeaway

CloudNativePG on k3s: PostgreSQL That Survives Node Failures

Why Not Just StatefulSets?

Installing CNPG

The Cluster Definition

The Backup Sidecar

Failover in Practice

Monitoring CNPG

The Trade-off

Traefik Rate Limiting That Actually Works on k3s

The Scrape Storm

The RateLimit Middleware

Applying It Per Service

Whitelisting Internal Traffic

The 429 Response

Monitoring Rate Limits

Vault Configuration Under Terraform: Staged Migration Without Downtime

The Scope Boundary

The Six Stages

Stage 1: Policies (lowest risk)

Stage 2: KV v2 Secrets Engine Mount

Stage 3: Kubernetes Auth Method

Stage 4: Kubernetes Auth Roles

Stage 5: Additional Auth Methods (deferred)

Stage 6: App Policies (pure additive)

Why Not Just vault write Everything?

The Maintenance Window

What Stays Manual Longest

Velero Backs Up to the Cluster It's Backing Up: Breaking the Backup Circularity

The Two-Problem Stack

Problem 1: Backup Circularity

Problem 2: Torn-State Hazard

The Four Options

Option A: Consolidate meta + data on one volume

Option B: External off-cluster target (Cloudflare R2)

Option C: PBS as the primary recovery layer

Option D: Combine B + C (what I chose)

Why Not Just `vault write` Everything?