DEV Community: david

My Terraform Runner Destroyed Itself Mid-Apply

david — Mon, 03 Aug 2026 09:54:57 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

On 2026-07-04, Atlantis applied a Terraform change that shut down the VM it was running on. The Terraform runner destroyed its own runtime environment mid-apply.

This is the story of a circular dependency that doesn't show up in terraform plan, why bpg/proxmox can't always update attributes in-place, and how moving Atlantis from a k3s Deployment to a dedicated LXC container eliminated the hazard entirely.

View the complete homelab infrastructure source on GitHub 🐙

The Setup

Atlantis was deployed as a k3s Deployment in the apps namespace, running on one of three k3s VMs managed by the same Proxmox Terraform stack. The Proxmox Terraform configuration (terraform/stacks/proxmox/) defines all three k3s VMs, their CPU, memory, disk, and boot settings.

The circular dependency: Atlantis runs on k3s VMs → Terraform manages k3s VMs → Atlantis applies Terraform changes to k3s VMs.

In practice, this was safe for most changes. memory, disk.size, cpu.cores — these can all be updated in-place by bpg/proxmox without stopping the VM. But certain attributes require a full VM shutdown-and-restart cycle. Specifically: cpu.units (scheduling priority) and serial_device configuration.

The Kill Shot

The Terraform diff for a cpu.units change looks like this:

resource "proxmox_virtual_machine" "vm_srv_k3s_11" {
  # ...
  cpu {
    units = 2048  # 2x scheduling priority over LXCs
  }
}

When bpg/proxmox detects a change to cpu.units, it can't hot-apply it. The provider issues a qmshutdown via the Proxmox API, waits for the VM to stop, applies the change, then starts the VM again.

On 2026-07-04, a PR changed cpu.units on one of the k3s VMs. Atlantis picked up the PR, ran terraform plan, showed the diff (in-place update), and ran terraform apply. The bpg/proxmox provider sent qmshutdown to the Proxmox API for the VM running the Atlantis pod.

The VM shut down. The Atlantis pod was killed. The Terraform apply was interrupted mid-execution. Kubernetes rescheduled the pod on a different node, but the apply state was lost.

The same thing happened again on a serial_device attribute change — another attribute that requires a VM shutdown. Two occurrences, same root cause: the Terraform runner was managing the infrastructure it depended on for its own execution.

Why `terraform plan` Doesn't Catch This

The circular dependency isn't expressed in the Terraform configuration. Atlantis's Pod spec doesn't reference the Proxmox VMs, and the Proxmox VMs don't reference Atlantis. Terraform sees two independent resource graphs. The dependency is physical, not declarative — Atlantis runs on the VMs, but Terraform doesn't know that.

terraform plan shows "will update in-place" for cpu.units. It doesn't know that "update in-place" means "shut down the VM first." That behavior is a provider implementation detail, not something Terraform's planning phase understands.

The lifecycle.ignore_changes block can prevent specific attributes from being planned, but that's a workaround, not a fix. You'd be ignoring a real change to avoid a structural hazard.

The Fix: Move Atlantis Off k3s

The fix was ADR-012: move Atlantis from a k3s Deployment to a dedicated LXC container (ct-srv-atlantis-01, VMID 204, IP 10.0.20.250).

# terraform/stacks/proxmox/lxc.tf
resource "proxmox_virtual_machine" "ct_srv_atlantis_01" {
  # Dedicated LXC — NOT managed by the same Terraform stack
  # Atlantis manages OTHER stacks, but its own container is outside the scope
  vm_id   = 204
  name    = "ct-srv-atlantis-01"
  node_name = "pve"
  # ...
}

The key difference: the Atlantis LXC is still defined in the Proxmox Terraform stack, but it's never managed by Atlantis itself. The atlantis.yaml repo config whitelists only specific repos and directories:

repos:
  - name: github.com/dwoitzik/homelab-infrastructure
    allowed_overrides: [apply_requirements, delete_source_branch_on_merge]
    apply_requirements: [approved, mergeable]
    projects:
      - dir: terraform/stacks/network
        workspace: default
      - dir: terraform/stacks/cloudflare
        workspace: default
      - dir: terraform/stacks/garage
        workspace: default

Notice: terraform/stacks/proxmox/ is not listed. Atlantis can plan and apply network, cloudflare, and garage changes — but never Proxmox changes. Proxmox changes go through a separate review process, or I apply them manually after careful review.

This breaks the circular dependency structurally: Atlantis manages everything except the infrastructure it runs on.

The Ansible Layer

The Atlantis LXC runs via Ansible, not k3s:

# ansible/roles/atlantis/tasks/main.yml
- name: Deploy Atlantis via Docker Compose
  community.docker.docker_compose_v2:
    project_src: /opt/atlantis
    state: present

The custom Dockerfile includes the Proxmox self-signed CA cert (pve-root-ca.crt) so Atlantis can talk to the Proxmox API over HTTPS:

FROM ghcr.io/runatlantis/atlantis:v0.30.0
COPY pve-root-ca.crt /usr/local/share/ca-certificates/pve-root-ca.crt
RUN update-ca-certificates

The Proxmox API token, Cloudflare API token, and MikroTik credentials are all stored in Ansible Vault and injected via Docker Compose environment variables.

The Pattern

Any Terraform runner that manages the infrastructure it runs on has this hazard. In a cloud environment, it's less obvious because terraform apply against an Azure VM doesn't restart the VM — Azure handles in-place updates at the platform level. But the same structural dependency exists: an Atlantis instance running on an Azure VM that manages that VM's NSG, disk, or network interface.

The clean fix is always the same: the runner manages everything except itself. If that's not possible, lifecycle { ignore_changes } on attributes that trigger restarts is the minimum viable mitigation.

The same pattern applies to CI/CD runners in enterprise environments. A self-hosted GitHub Actions runner managing its own host's infrastructure via Terraform has the identical circular dependency. The fix is the same: separate the runner's infrastructure from the infrastructure it manages, even if they share the same cloud account.

The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab

david — Sat, 01 Aug 2026 00:55:59 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Three times in one week, my Proxmox host froze completely. No SSH, no web UI, no console. The only recovery was a hard power cycle. Each time, I lost every running VM and LXC simultaneously — k3s control plane, workers, databases, all of it.

The first time, I blamed a kernel bug. The second time, I blamed ZFS. The third time, I found the actual chain: a marginal PSU causing CPU throttling under load, which triggered a ZFS 2.4.1 ARC deadlock that locked the entire host.

View the complete homelab infrastructure source on GitHub 🐙

The Symptoms

The freezes weren't random. They correlated with sustained I/O load — specifically, when multiple workloads competed for NVMe bandwidth simultaneously. The pattern:

k3s etcd writing to the same NVMe as the ZFS pool
Ollama LLM inference spinning up large model loads
Paperless OCR bursts hitting the disk queue hard

Under moderate load, everything was fine. Under sustained high load across multiple subsystems, the host would lock up within minutes.

The first two freezes produced no useful logs. journalctl cut off mid-entry. No kernel panic, no oops, no MCE. Just silence.

The PSU Theory

The BMAX Mini PC* ships with a 90W PSU. Under normal operation, the Ryzen 7 5825U draws 15W TDP and the NVMe pulls another 8W. But when all cores are loaded — k3s workers, Ollama inference, OCR workers — the CPU can boost well above base TDP.

I added power monitoring via a TP-Link Tapo P110* to track actual draw:

Idle:              47W
k3s + normal apps: 68W
All cores loaded:  94W
Peak (all + NVMe): 108W

108W peak against a 90W PSU. The PSU was delivering more than its rated capacity, which means the 12V rail was sagging under load. When the rail sags below the CPU's brownout threshold, the CPU enters clock throttling to reduce power draw — and that's where the real trouble started.

The ZFS Deadlock

ZFS uses a Transaction Group (txg) commit cycle. Every few seconds, dirty data in the ARC (Adaptive Replacement Cache) gets flushed to disk in a txg commit. The commit is a synchronous operation — ZFS holds locks on dirty pages and waits for the I/O to complete before releasing them.

When the CPU throttles mid-commit, the I/O latency spikes because the NVMe controller shares the same power rail. The txg commit can't complete within its timeout window, ZFS holds the ARC locks, new I/O requests pile up waiting for those locks, and the entire storage stack deadlocks.

In ZFS 2.4.1 specifically, the ARC eviction path has a known issue where it can block on the same locks held by a stalled txg commit. This creates a circular wait: ARC eviction needs txg to complete, txg needs I/O bandwidth, I/O is delayed by CPU throttling, CPU throttling is caused by the PSU sag, and the PSU sags because everything is running at once.

The host doesn't panic. It just stops responding to anything that touches disk.

The Fixes

Fix 1: Disk I/O Isolation

The k3s VMs and the ZFS pool were sharing the same NVMe. QEMU's page cache was adding to the I/O contention. I moved the k3s VM disks to cache=none + aio=native:

# terraform/stacks/proxmox/vm.tf
disks {
  scsi {
    scsi0 {
      disk {
        size    = "50G"
        cache   = "none"
        aio     = "native"
        storage = "local-lvm"
      }
    }
  }
}

cache=none bypasses QEMU's page cache entirely — the guest handles its own caching. aio=native uses Linux's native AIO instead of the emulated thread pool, reducing context switching. This alone reduced I/O latency variance by 40% under load.

Fix 2: ZFS ARC Limits

Without caps, the ARC can grow to consume most of available memory, which means more dirty pages waiting for txg commit. I capped the ARC:

# /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=4294967296  # 4GB max ARC

On a 64GB host, 4GB for ARC is generous enough for read caching without creating a massive pool of dirty pages that can deadlock under I/O pressure.

Fix 3: txg Timeout Tuning

ZFS defaults to a 5-second txg timeout. On hardware with known I/O fragility, I shortened it:

# /etc/modprobe.d/zfs.conf
options zfs zfs_txg_timeout=2

A shorter timeout means ZFS gives up on a stalled commit sooner, which sounds counterintuitive, but it prevents the deadlock from propagating. The txg will retry on the next cycle with a fresh set of locks rather than holding the old ones indefinitely.

Fix 4: Staggered Boot Order

The boot storm was a separate but related trigger. All VMs and LXCs starting simultaneously spiked load to 147 on the single NVMe. I added staggered boot ordering:

# NFS boots first (no k3s dependency)
# k3s VMs boot 30s apart
# LXCs start last

This prevents the "everyone needs disk at once" pattern that was triggering the PSU sag in the first place.

What It Taught Me

The root cause wasn't software — it was hardware. But the software (ZFS 2.4.1) turned a power delivery problem into a host-level deadlock. On different storage (ext4, XFS), the same PSU sag would have caused I/O errors or performance degradation, not a complete freeze.

The lesson: when debugging host-level freezes, check the physical layer first. PSU capacity, thermal throttling, NVMe heat — before diving into kernel logs and ZFS tuning parameters. The second lesson: ZFS is brilliant for data integrity, but its complexity means edge cases exist where a minor hardware issue becomes a major software deadlock.

After these four changes, the host has been stable through sustained load tests. The PSU is still marginal — a proper fix would be a higher-capacity unit — but the software mitigations prevent the hardware weakness from cascading into a deadlock.

Power delivery and I/O isolation are the same problem at enterprise scale: Azure VMs have their own power and I/O profiles, Premium SSD P30 IOPS limits interact with CPU burst credits, and a misconfigured VM series can trigger the same "everything competes for one resource" pattern that killed my homelab. The difference is that Azure hides the PSU behind an SLA — you just get a throttled VM instead of a frozen host.

How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead

david — Mon, 27 Jul 2026 10:25:30 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

Velero's nightly backup was taking 207 seconds and backing up 33.4GB. That's normal for a cluster with 15+ PVCs — except when one volume accounts for 65% of the total bytes. The nfs-provisioner's root-mount volume was backing up the entire shared NFS export as a single blob, even though every app's individual PVC (a subdirectory of the same tree) was already backed up separately.

The fix was a one-line annotation. The investigation almost went wrong when the first hypothesis pointed at the wrong disk entirely.

The Symptom

Velero's backup stats showed a disproportionate volume:

PodVolumeBackup: nfs-provisioner-root
Namespace: nfs-provisioner
Size: 33.4GB
Duration: 207s

Total backup for that run: ~51GB. This single volume was 65% of the total. Every other PVC was 1-4GB.

The Investigation

The nfs-provisioner pod mounts the entire shared NFS export at /persistentvolumes. Every app's PVC is a subdirectory of this tree. Velero's defaultVolumesToFsBackup: true setting means every volume the pod mounts gets backed up — including the root mount that contains all the subdirectories.

Each app's own pod also mounts its PVC subdirectory separately. Velero backs that up too. So the same data is backed up twice: once as part of the root mount, once as the individual PVC.

The First Hypothesis (Wrong)

My first guess was garage-data/immich-library — Immich's photo library, which lives on separate USB-attached disks (Seagate Portable 2TB External HDD*). Excluding it would have reduced backup size, but the USB disks aren't on the contended rpool. The backup bottleneck was the SSD, not USB. Excluding the Immich volume would have been the wrong fix for the right symptom.

The Right Fix

Verified the target before acting:

# Confirm the root-mount volume targets rpool (the contended disk)
findmnt /persistentvolumes
# /dev/zvol/rpool/nfs  ← confirmed rpool

# Confirm the individual PVCs are subdirectories of the same tree
ls /persistentvolumes/
# default-elasticsearch-data-0/
# default-immich-redis-data-0/
# default-minecraft-data/
# ... (each is a separate PVC)

The root mount is the entire rpool. Every individual PVC is a subdirectory. Backing up the root mount redundantly backs up everything.

The Exclusion

# kubernetes/system/nfs-provisioner/application.yml
podAnnotations:
  backup.velero.io/backup-volumes-excludes: nfs-subdir-external-provisioner-root

One annotation. Velero skips the root-mount volume entirely. Every app's individual PVC still gets backed up through its own pod's mount.

Result

Before: 51GB total, 33.4GB from root mount (65%)
After:  17.6GB total, 0GB from root mount

Backup duration dropped from 207s to ~60s. No data was lost — every PVC is still backed up individually.

The Pattern

This is a common Velero gotcha with NFS provisioners:

NFS provisioner pod mounts the entire export root
App pods mount individual PVC subdirectories
Velero backs up both — root mount AND individual PVCs
The root mount is redundant because it contains everything the individual PVCs contain

The fix is always the same: exclude the root mount via backup.velero.io/backup-volumes-excludes.

How to Find Redundant Backups

# List all PodVolumeBackups and their sizes
kubectl get podvolumebacks -A -o json | \
  jq -r '.items[] | "\(.spec.volume) \(.status.progress.totalBytes // 0) \(.metadata.namespace)"' | \
  sort -k2 -rn | head -10

If one volume is disproportionately large and its data is already covered by individual PVC backups, it's redundant.

🚀 Azure Firewall - Enterprise Forced Tunneling Edition — €49

Cycle-error-free resource ordering - deploys first time, every time
KMS & Azure AD bypass routes - no broken Windows VMs or auth failures
Dynamic for_each subnet binding - scales to any number of Spokes
IP Group-based firewall policies - no hardcoded IP addresses
FQDN baseline rules for Windows Updates and core Microsoft services

Get the enterprise module →

Full source code · one-time payment · instant download

Velero's defaultVolumesToFsBackup: true is convenient but dangerous with NFS provisioners. The root mount contains everything, and Velero doesn't know it's redundant. Check your PodVolumeBackup stats — if one volume is 65% of your total, you're backing up the same data twice.

Why a Cloud Backup Sync Was Failing Daily for Six Weeks

david — Mon, 27 Jul 2026 10:08:19 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

A systematic ansible-playbook --check --diff sweep across every host group turned up a stale Vault credential and a cron job that appeared to have simply stopped running. The initial read was a straightforward drift problem: fix the credential, restore the cron schedule, move on. That read was wrong on both the diagnosis and, as it turned out one PR later, the intended fix.

View the complete homelab infrastructure source on GitHub 🐙

What the Audit Sweep Found

Two issues on the PBS management host, both real:

1. A stale Vault credential that would have clobbered a working live token. The pbs_rclone_gdrive_token stored in Vault was from April; the live rclone.conf on the host had a June token - rclone had refreshed it itself as OAuth tokens do, and Vault had never been updated to match. Running the Ansible role for real (not just --check) would have overwritten the working, self-refreshed credential with a four-month-stale one, breaking authentication.

# Confirmed: live rclone.conf token newer than Vault's stored value
diff <(rclone config show pbs-gdrive | grep token) \
     <(vault kv get -field=pbs_rclone_gdrive_token secret/pbs)
# Fixed: update Vault to match the live, working token
vault kv patch secret/pbs pbs_rclone_gdrive_token="$(rclone config show pbs-gdrive | grep token)"

2. The offsite sync cron job simply wasn't on the host. Its own log file, pbs-to-gdrive.log, showed it running - mostly failing - from May 4 through June 14, then nothing. No entries at all after that date. Something had stopped the schedule.

The Wrong First Conclusion

The natural read of "cron job disappeared, logs show it was mostly failing before that" is: something broke it, and the failures were probably why. Re-enable the schedule, fix whatever was causing the failures, done.

Actually diagnosing the failures required running a real, bounded manual sync rather than reasoning from the log lines alone:

rclone sync /mnt/pbs-datastore gdrive:pbs-backup --transfers 4 --checkers 8 \
  --stats 30s --stats-one-line --max-duration 30m
# Transferred: 3.2 MiB / ~2.1 TiB, 0%, 1.6 KiB/s, ETA 12 weeks

1.6 KiB/s. Not a credential failure, not a network failure - Google Drive's API was throttling the sync hard, and the reason became clear from what PBS actually stores on disk: not a handful of large archive files, but tens of thousands of small deduplicated chunk files. Google Drive's API rate-limits per-file operations aggressively, and a chunk-based datastore is close to the worst-case access pattern for that limit.

At an observed throughput of ~1.6 KiB/s against a multi-terabyte datastore, the estimated time for just the initial full sync was on the order of 12 weeks. The daily cron job, scoped to a 24-hour window, could never have completed a first full sync - every single run for six weeks was doomed from the start, not by a bug in the job but by a structural mismatch between the storage format and the destination's API characteristics.

The Fix That Turned Out to Be Wrong

The first PR treated the missing cron job as accidental drift and restored it - re-added the scheduled task, assuming it had been silently dropped by some unrelated config change and needed putting back.

It hadn't been silently dropped. Checking with the account owner surfaced the actual story: the cron job had been turned off deliberately, because the destination Google Drive account doesn't have enough free storage quota for the PBS datastore's full size. Turning the schedule back on wasn't fixing drift - it was un-doing an intentional decision that the earlier PR had mistaken for an accident.

# What the first PR did (wrong - re-enables a deliberately-disabled job)
- name: Sync PBS offsite cron
  ansible.builtin.cron:
    name: pbs-to-gdrive-sync
    state: present   # ← reintroduces a schedule the user turned off

# The corrected version
- name: Remove PBS offsite cron (deliberately disabled -- insufficient quota)
  ansible.builtin.cron:
    name: pbs-to-gdrive-sync
    state: absent

The correction PR also removed the live crontab entry that the first "fix" had reintroduced, and rewrote the documentation to describe the situation accurately: not "broken, needs re-enabling" but "deliberately deferred pending a decision about storage quota or a different offsite strategy."

The Same Log Pattern, Two Opposite Conclusions

The interesting part isn't the throttling finding by itself - it's that the same evidence (pbs-to-gdrive.log showing failures, then silence) supported two completely different, mutually exclusive conclusions:

"This is drift - something broke, restore the intended state."
"This is intentional - a human made a deliberate decision that the automation shouldn't override."

Ansible's entire idea of "correct state" assumes the desired state is knowable from the playbook. It has no way to distinguish "this differs from the playbook because of an accident" from "this differs from the playbook because a human changed their mind after the playbook was written." Both look identical in a --diff output. The only way to tell them apart is to ask.

This is also, notably, the same false-positive class that closed out a side finding in the same PR: an AdGuard admin-password "drift" flagged by the same sweep turned out to be bcrypt's per-render random salt making two hashes of the same password look different in a diff - not a real config drift either. Two findings from one audit sweep, both initially read as drift, both actually non-issues once verified against ground truth rather than assumed from the diff output.

What This Argues For

Any Ansible task that can toggle a schedule, feature flag, or setting on or off needs a documented reason when it's off - not just an inferred one from "the playbook says it should be on." A short comment (# disabled 2026-06-XX: destination quota insufficient, see docs/backup-strategy.md) turns a future --diff output from "ambiguous, ask the owner" into "documented, already explained."

And more broadly: a drift-detection sweep is good at finding that something differs from the intended state. It says nothing about why it differs, and treating every diff as an accident to be corrected is exactly as wrong as treating every diff as intentional and ignoring it. Both directions of this mistake happened in the same PR chain here, within hours of each other.

The direct Azure parallel: Azure Policy remediation tasks that "fix" a resource back to policy-compliant state can just as easily undo a deliberate, documented exception a team made for a legitimate reason. Policy exemptions exist for exactly this - but only if someone actually creates the exemption object instead of leaving the deviation undocumented and hoping nobody re-applies the policy. An undocumented intentional deviation and an actual accidental drift look identical to any automated compliance sweep.

MikroTik vs Ubiquiti for Home Network 2026

david — Sun, 19 Jul 2026 11:07:31 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

View the complete homelab infrastructure source on GitHub 🐙

This is the networking debate that never dies in homelab circles: MikroTik or Ubiquiti? I have run both in production — MikroTik RB5009 as my primary router for over two years with every firewall rule and VLAN managed via Terraform, and Ubiquiti UniFi gear in a friend's lab where I helped with deployment. This is not a spec-sheet comparison. It is a practical breakdown of what each platform actually delivers for a homelab that runs Proxmox, K3s, and zero-trust networking.

TL;DR — The Verdict

MikroTik RB5009 if you want maximum control, Terraform-native automation, and the best price-to-performance in router hardware. The learning curve is steep, but the ceiling is infinitely higher.

Ubiquiti UniFi Dream Router if you want plug-and-play setup, a polished UI, and an ecosystem where access points, switches, and cameras all manage from one dashboard. You pay a premium and give up depth of control.

The short version: MikroTik is for engineers who want to own their network stack. Ubiquiti is for users who want a network that works without deep networking knowledge. Both are valid — but for a homelab running infrastructure-as-code, MikroTik wins decisively.

Quick Comparison

Feature	MikroTik RB5009	Ubiquiti UniFi Dream Router
Price	~€180*	~€300*
CPU	Marvell 98DX3236 (ARM)	Quad-core ARM Cortex-A57
RAM	1GB	2GB
Ports	7x GbE + 1x 10G SFP+	4x GbE + 1x 2.5GbE WAN
Switch Chip	Yes (hardware offloading)	Integrated (software)
OS	RouterOS 7	UniFi Network
Management	CLI, WinBox, API, Terraform	UniFi Controller (UI)
VLAN Support	Full (bridge VLAN filtering)	Full (VLAN networks)
VPN	WireGuard, OpenVPN, IPsec, PPTP	WireGuard, OpenVPN, L2TP
Wi-Fi	None (bring your own AP)	Built-in Wi-Fi 6
Price/Performance	Excellent	Moderate

Category-by-Category Breakdown

Design & Build Quality

The RB5009 is an industrial-looking metal box. No Wi-Fi, no antennas, no RGB. It has 7 Gigabit Ethernet ports, one 10G SFP+ cage, and a USB 3.0 port. The metal chassis dissipates heat passively — no fan, no noise. It is designed to live in a rack or on a shelf without anyone noticing it. At 100x155x33mm, it is compact enough to mount in a 10-inch rack (I use a DIGITUS 10" rack* for mine).

The UniFi Dream Router (UDR) is designed to be seen. It is a white, cylindrical unit with a built-in Wi-Fi 6 access point and a small LED status ring. It looks like something from a modern living room, not a server closet. The build quality is good — solid plastic, well-ventilated — but it is designed for desk or shelf placement, not rack mounting. The form factor says "consumer product" while the RB5009 says "networking equipment."

Winner: MikroTik RB5009 for homelab use. The fanless design, rack-friendly form factor, and 10G SFP+ port make it the better infrastructure component. The UDR wins for living room aesthetics if that matters to you.

Features & Flexibility

This is where the gap becomes a canyon.

RouterOS 7 on the RB5009 is, functionally, a full Linux networking stack exposed through a proprietary CLI and API. You get: bridging with VLAN filtering, firewall with stateful packet inspection, NAT (source and destination), QoS (HTSQ, PCQ, CAKE), MPLS, BGP, OSPF, VPLS, bonding, MLAG, traffic shaping, bandwidth limiting, DHCP server with static bindings, DNS forwarding with split-horizon, CAPsMAN (centralized AP management), and more. The feature set is closer to enterprise networking gear (Cisco, Juniper) than to consumer routers.

The UDR runs UniFi Network, which is a software-defined networking platform. It provides: VLAN segmentation, firewall rules, IDS/IPS (Intrusion Detection/Prevention), traffic identification, guest portal, and DPI (Deep Packet Inspection). The UniFi Controller — a web application that runs either on the UDR itself or on a separate server — manages all of it through a graphical interface.

The UniFi feature set is impressive for what it is, but it is fundamentally a curated subset. You get the features Ubiquiti decided to implement, configured the way Ubiquiti decided to configure them. MikroTik gives you the entire toolbox and lets you build whatever you want. For a homelab that runs Terraform-managed infrastructure, the difference is critical — MikroTik's API and CLI let you automate everything; UniFi's API is limited and unofficial.

Winner: MikroTik RB5009 for depth and automation potential. UniFi UDR for breadth of managed features out of the box (IDS/IPS, DPI, traffic identification).

Ease of Use

Ubiquiti wins this category flatly, and it is not close.

The UniFi Controller provides a visual dashboard showing network topology, connected clients, traffic flows, and security events. VLAN creation is a form fill. Firewall rules are a guided wizard. Firmware updates are one-click. If you have never configured a router before, the UDR will have your network segmented and secured in under 30 minutes.

RouterOS is a different world. The WinBox GUI exists and works, but it mirrors the CLI structure — you navigate menus organized by protocol and feature, not by task. Creating a VLAN involves: creating a bridge, adding ports to the bridge, creating VLAN entries on the bridge, creating an interface for the VLAN, assigning an IP address, creating a DHCP server, and creating firewall rules for the new network. Each step is a separate resource. The CLI (or Terraform) makes this repeatable; the GUI makes it tedious.

I will not sugarcoat it: RouterOS has a steeper learning curve than any consumer router OS. The first time you configure Bridge VLAN Filtering, you will lock yourself out. The second time, you will not. By the third time, you will understand L2 networking better than 99% of home network users. That knowledge has value beyond MikroTik — it transfers to every networking platform.

Winner: Ubiquiti UDR for initial setup and ongoing management. MikroTik for users who want to actually understand their network.

Security & Firewall

Both platforms support stateful packet inspection, NAT, and rule-based firewalling. The differences are in philosophy and depth.

MikroTik gives you full control over every firewall rule. The RouterOS firewall is a chain-based system (input, forward, output) with rule ordering that matters. You can filter by source/destination IP, port, protocol, connection state, interface, VLAN, packet content, and more. The zero-trust approach I documented in MikroTik Zero Trust Firewall with Terraform is built entirely on this: default-drop on input and forward, with explicit allow rules for each service.

Here is the actual Terraform code that creates the final drop rule in my firewall:

resource "routeros_ip_firewall_filter" "fwd_99_drop_all" {
  action  = "drop"
  chain   = "forward"
  comment = "99: Global - Final Drop (Zero Trust Policy)"
}

That single rule, placed last in the chain, enforces zero trust for all forwarded traffic. Every allowed service must have an explicit rule before it. The simplicity is deceptive — the complexity is in the rules above it, but the principle is clear: deny everything, allow by exception.

Ubiquiti offers IDS/IPS as a built-in feature, which MikroTik does not have natively. The UniFi IDS/IPS inspects traffic for known attack signatures and can automatically block or alert on suspicious activity. For a home network, this provides a layer of protection that a MikroTik firewall alone does not — unless you add Suricata or Snort as a separate IDS on a mirror port.

The tradeoff: UniFi's IDS/IPS is a black box. You cannot customize the signatures, you cannot tune the sensitivity beyond broad categories, and you cannot export the logs to a SIEM easily. MikroTik's firewall is transparent — every rule is visible, every log entry is configurable, and you can pipe logs to an external syslog server or Loki stack for analysis.

Winner: Tie. MikroTik wins for transparent, automatable, zero-trust firewall rules. UniFi wins for built-in IDS/IPS that requires zero configuration. The ideal setup uses both — MikroTik as the router/firewall, UniFi APs for wireless, and an external IDS like Suricata for deep packet inspection.

VPN

Both platforms support WireGuard and OpenVPN. The implementation differences matter.

MikroTik RB5009 runs WireGuard natively in RouterOS 7. The WireGuard implementation supports multiple peers, fine-grained allowed IPs, and persistent keepalive — everything you need for site-to-site or remote access VPN. I run WireGuard on my RB5009 for remote access to the homelab, and the Terraform code for it is in the repo:

resource "routeros_interface_wireguard" "wg0" {
  listen_port = 13231
  name        = "wg0"
}

MikroTik also supports IPsec (IKEv1 and IKEv2) for site-to-site VPN with other vendors, OpenVPN for legacy client access, and PPTP (which you should never use). The RB5009's CPU handles WireGuard encryption in software — it maxes out around 1.5-2Gbps throughput, which is more than enough for a home WAN link.

UniFi UDR supports WireGuard and OpenVPN through the controller interface. Setup is simpler — a few form fills and you have a VPN tunnel. The UDR also supports Teleport (Ubiquiti's proprietary VPN) which uses WireGuard under the hood but with a simplified peer exchange mechanism.

The limitation: UniFi's VPN integration is shallow. You cannot fine-tune WireGuard parameters (MTU, pre-shared keys, handshake intervals) the way you can on MikroTik. Site-to-site VPN between UniFi and non-UniFi devices is possible but not well-documented. For a homelab that needs VPN access to specific VLANs or subnets, MikroTik's granular control is essential.

Winner: MikroTik RB5009 for VPN flexibility and fine-grained control. UniFi wins for quick-and-easy remote access setup.

VLAN Support

Both platforms support VLAN tagging, trunk ports, and access ports. The implementation approach is fundamentally different.

MikroTik handles VLANs through Bridge VLAN Filtering on a single bridge. All physical ports join one bridge, and VLANs are defined as entries in the bridge's VLAN table. This approach enables hardware offloading on the RB5009's switch chip — tagged traffic is switched in hardware, not software. The result: full wire-speed VLAN performance even with dozens of VLANs configured.

The Terraform code for my entire VLAN matrix — five VLANs, trunk ports, access ports, IP addresses, and DHCP servers — is about 100 lines of HCL. Add a new VLAN? Change one map and run terraform apply.

UniFi handles VLANs by creating "VLAN Networks" in the controller. Each VLAN is a separate network object with its own subnet, DHCP scope, and firewall rules. Trunk ports are configured per-switch-port in the UI. The approach is intuitive — each VLAN is a first-class object in the UI — but it ties you to the UniFi Controller for management.

Both approaches work well. MikroTik's is more flexible and automatable. UniFi's is more visual and self-documenting. For a homelab that needs VLANs for network segmentation (management, server, DMZ, IoT, admin), both platforms handle it without issue.

Winner: Tie. MikroTik for automation and hardware offloading. UniFi for visual management and simplicity.

Terraform & Automation Support

This is where MikroTik pulls away decisively for any homelab running infrastructure-as-code.

The MikroTik Terraform Provider (routeros) is mature, well-maintained, and covers nearly every RouterOS resource: interfaces, bridges, VLANs, firewall rules, DHCP, DNS, WireGuard, routing, and more. You can define your entire network stack in HCL, version it in Git, and apply changes through a CI pipeline or Atlantis. This is exactly how I manage my homelab — every MikroTik configuration change goes through a PR, gets reviewed, and is applied via Atlantis.

The Ubiquiti Terraform Provider (ubiquiti) is unofficial and limited. The community provider (paultyng/ubiquiti) covers basic device management but does not support network configuration, VLAN creation, or firewall rules through Terraform. There is no official UniFi Terraform provider from Ubiquiti. This means every network configuration change on UniFi must go through the controller UI — there is no IaC path.

For a homelab that already uses Terraform for Proxmox VMs, MikroTik firewall rules, and Kubernetes resources, the lack of UniFi Terraform support is a dealbreaker. You would end up with two configuration workflows: Terraform for everything except the network, and manual UI clicks for the network. That is exactly the kind of configuration drift that IaC is designed to prevent.

Winner: MikroTik RB5009 — decisively. No meaningful competition in this category.

Ecosystem & Expandability

Ubiquiti wins on ecosystem breadth. The UniFi ecosystem includes switches, access points, cameras (Protect), phones (Talk), displays (Connect), and a centralized controller that manages all of it. If you want one vendor for your entire network stack — router, switches, APs, cameras — UniFi provides that experience. The ecosystem lock-in is real, but so is the integration quality.

MikroTik has a broader product line than most people realize — routers, switches, access points, antennas, and even outdoor wireless gear. CAPsMAN (Centralized Access Point Manager) lets you manage MikroTik APs from the router itself. But the ecosystem is not as tightly integrated as UniFi. Each device runs RouterOS independently; there is no single controller dashboard (The Dude exists but is not comparable to the UniFi Controller).

For a homelab that primarily needs a router and a few switches, MikroTik's ecosystem is sufficient. For a whole-home networking deployment with cameras, multiple APs, and managed switches, UniFi's ecosystem provides a more cohesive experience.

Winner: Ubiquiti UDR for integrated ecosystem. MikroTik for standalone router/switch deployments.

Price & Value

The RB5009 costs roughly €180. The UniFi Dream Router costs roughly €300. That €120 difference buys you: built-in Wi-Fi 6 on the UDR (MikroTik requires a separate AP, ~€50-100 for a hAP ac3 or cAP ax), the UniFi Controller software, and IDS/IPS.

If you add a MikroTik AP to the RB5009 (which you will need if you want Wi-Fi), the total cost approaches the UDR. But you get a significantly more powerful router and a separate, dedicated AP — which is arguably better than a combined router/AP unit that must serve both roles.

For a homelab that does not need Wi-Fi from the router (most homelabs have a separate AP or run wired-only), the RB5009 at €180 is the clear value winner. For a whole-home setup where one device handles routing and Wi-Fi, the UDR's combined value is harder to beat.

Winner: MikroTik RB5009 for homelab use. UDR for whole-home Wi-Fi + routing in one device.

Who Should Buy Which

Buy the MikroTik RB5009 if:

You run Terraform for infrastructure management and want network automation
You want zero-trust firewall rules with full control over every packet
You value learning networking over having it pre-configured
You do not need Wi-Fi from the router itself
Budget matters — €180 buys enterprise-grade routing

Buy the UniFi Dream Router if:

You want plug-and-play setup with a polished UI
You need built-in Wi-Fi 6 without a separate AP
You want IDS/IPS without deploying a separate appliance
You are building a whole-home UniFi ecosystem (switches, APs, cameras)
You do not want to learn CLI-based network configuration

Buy both (MikroTik router + UniFi AP) if:

You want the best of both worlds: MikroTik routing/firewall automation with UniFi wireless management
This is what I would do if I were starting a new homelab today. MikroTik for the router, UniFi for the APs. The MikroTik handles VLANs, firewall, and VPN via Terraform. The UniFi AP handles wireless through its controller. Two vendors, two strengths, zero compromise.

Frequently Asked Questions

Can MikroTik and UniFi devices coexist on the same network?

Yes, they work together without issue. MikroTik handles routing and VLAN segmentation; UniFi APs broadcast the VLAN-tagged SSIDs. The key configuration point: the switch port connecting the UniFi AP must be a trunk port carrying all VLAN SSIDs. This is configured on the MikroTik side (bridge VLAN filtering) and the UniFi side (port profile with VLAN tagging). Both platforms handle this standard configuration well.

Is MikroTik really harder to set up than Ubiquiti?

Yes, initially. MikroTik's RouterOS has a learning curve that takes 2-4 weeks to climb for basic proficiency. Ubiquiti's UniFi can be configured in an afternoon. But the long-term payoff is significant: once you understand RouterOS, you can configure any networking scenario without vendor lock-in. The knowledge transfers to Cisco, Juniper, and every other networking platform. UniFi knowledge stays within the UniFi ecosystem.

Should I use MikroTik for my home Wi-Fi?

Not directly. MikroTik makes access points (hAP ac3, cAP ax), but their wireless performance and roaming are inferior to UniFi's. MikroTik excels at routing and switching; let a dedicated AP handle wireless. Pair an RB5009 with a UniFi U6 Lite or U6 Pro for the best of both worlds.

How does MikroTik security compare to Ubiquiti?

MikroTik's firewall is more configurable but requires manual hardening. RouterOS 7 has had security vulnerabilities (WinBox CVE in 2018, Chimay Red attacks), but MikroTik patches quickly and the attack surface is manageable with proper firewall rules and by disabling unused services. UniFi's IDS/IPS provides automated threat detection that MikroTik lacks natively, but the black-box nature of the detection limits its utility for security-conscious users who want full visibility.

Can I manage MikroTik with Terraform?

Yes, the routeros Terraform provider is mature and actively maintained. It covers interfaces, bridges, VLANs, firewall rules, DHCP, DNS, WireGuard, routing, and more. I manage my entire MikroTik RB5009 configuration through Terraform — every change goes through Git, gets reviewed, and is applied via Atlantis. The provider is at version 1.x and supports RouterOS 7.x. See the Terraform VLAN filtering article for a real-world example.

What is the best MikroTik router for a home lab?

The RB5009 is the sweet spot. It has 7 Gigabit ports (enough for most homelabs), a 10G SFP+ uplink for future-proofing, a switch chip for hardware-offloaded VLANs, and RouterOS 7 with full feature support. The hEX S (€70) is the budget option but lacks the 10G port and has a less capable switch chip. The CCR2004 (€300+) is overkill for home use unless you need 10Gbps routing throughput.

Conclusion

This is not a close comparison — but it is not a one-sided one either.

MikroTik wins for any homelab running infrastructure-as-code. The Terraform integration, the granular firewall control, the hardware-offloaded VLAN switching, and the 10G SFP+ uplink make the RB5009 the definitive choice for engineers who want to own their network stack. At €180, it is also the better value for routing performance.

Ubiquiti wins for whole-home networking where ease of use, ecosystem integration, and built-in Wi-Fi matter more than IaC automation. The UniFi Controller is the best consumer networking management platform available, and the ecosystem of switches, APs, and cameras is genuinely compelling.

For a homelab running Proxmox, K3s, and Terraform: buy the MikroTik RB5009. The learning curve pays for itself in flexibility, control, and the ability to manage your network the same way you manage your servers — as code.

🚀 Azure Zero-Trust Starter Pack — €99

Includes: Acmebot Enterprise VNet, Hub & Spoke Zero-Trust, Azure Firewall Forced Tunneling
Save €48 vs. buying the three modules separately
Bonus: ISO 27001 Auditor Checklist (PDF) - Annex A control map for all 3 modules
Everything to pass your first Azure compliance audit
Full source code for all 3 modules - no lock-in, no black box

Get the enterprise module →

Full source code · one-time payment · instant download

Best NAS for Home Backup 2026

david — Sun, 19 Jul 2026 10:53:20 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

View the complete homelab infrastructure source on GitHub 🐙

A NAS is the single most important piece of hardware in any homelab. I learned this the hard way when a Proxmox host's NVMe died without a verified backup — the incident I wrote about in Backup & Restore: The Test You Never Verified. Since then, I run the 3-2-1 rule religiously: three copies of critical data, on two different media types, with one offsite. A NAS is the backbone of that strategy.

But the NAS market in 2026 is crowded. Synology still dominates mindshare, QNAP competes on features, Terramaster fights on price, and UGREEN is the new player trying to disrupt both. This comparison is based on real deployment experience and hands-on testing — not spec sheet comparisons.

TL;DR — The Picks

Model	Bays	CPU	RAM	Price (approx.)	Best For
Synology DS224+*	2	Intel Celeron J4125	2GB (expandable)	~€300	Best Overall
QNAP TS-264*	2	Intel Celeron N5095	8GB	~€350	Best Features
Terramaster F2-223*	2	Intel Celeron N4505	4GB	~€200	Best Budget
Synology DS423+*	4	Intel Celeron J4125	2GB (expandable)	~€500	Best for Growth
UGREEN DXP2800*	2	Intel N100	8GB	~€280	Best Newcomer

If you want the short version: the Synology DS224+ remains the default recommendation for anyone who wants a NAS that works reliably for 5+ years with minimal maintenance. If you need more bays and growth potential, the Synology DS423+ is the investment pick. If budget matters and you are comfortable with less polish, the Terramaster F2-223 punches well above its price.

Selection Criteria — What Matters in a Home Backup NAS

A NAS for home backup has different priorities than a NAS for media streaming or enterprise storage:

Reliability over features — You need a device that stays online 24/7, handles power loss gracefully, and does not corrupt your data during firmware updates. Synology leads here by a wide margin. QNAP's feature set is broader but their update track record is rougher.

Disk compatibility — Not all NAS enclosures work well with all drives. Synology's drive compatibility list is restrictive (they now push their own HAT5300 drives), but this conservatism means fewer surprise failures. QNAP and Terramaster are more permissive.

Backup software — The native backup application matters more than the hardware. Synology Active Backup for Business, Hyper Backup, and Snapshot Replication are best-in-class for consumer NAS. QNAP's Hybrid Backup Sync is powerful but more complex. Terramaster and UGREEN's software ecosystems are younger and less proven.

RAID support — For 2-bay units, RAID 1 (mirroring) is the minimum. For 4-bay, RAID 5 or RAID 6 gives you redundancy with capacity. Synology's SHR (Synology Hybrid RAID) is particularly clever — it lets you mix drive sizes without wasting space, which matters when you upgrade drives incrementally.

Power consumption — The NAS runs 24/7. An idle draw of 10W versus 25W is significant over a year. Disk count matters more than the enclosure itself — two 7200RPM HDDs draw more power than the entire NAS unit.

Expandability — RAM upgrades, NVMe cache drives, 10GbE expansion, and USB connectivity for external backup targets all extend the useful life of a NAS.

Individual Reviews

1. Synology DS224+ — Best Overall

2-Bay · Intel Celeron J4125 · 2GB DDR4 (expandable to 6GB) · ~€300

The DS224+ is the NAS I recommend to everyone who asks, and the one I run myself. It is not the cheapest, not the fastest, and not the most feature-rich — it is the one that works without drama for years.

Synology's DSM 7.2 operating system is the reason to buy this NAS. The interface is polished, the backup applications are mature, and the package ecosystem (Docker support, surveillance, media serving) covers every use case without feeling bloated. Hyper Backup handles encrypted, deduplicated backups to local drives, remote Synology NAS, and cloud targets (S3, Backblaze B2, Google Drive) with a single application.

The J4125 CPU is not fast — it is a 4-core embedded chip designed for low power, not performance. For file serving, backup, and even Docker containers running lightweight services, it is perfectly adequate. For Plex transcoding of 4K HEVC content, it will struggle. The 2GB base RAM is tight if you run multiple packages; upgrading to 6GB via the SO-DIMM slot is essential if you use Docker or run more than 2-3 services.

Power consumption is excellent: idle around 10-12W with two drives spun down, 18-22W under active load. This matters for a device running 24/7.

Pros:

DSM 7.2 — best NAS operating system available
Excellent power efficiency
Hyper Backup and Active Backup are mature, reliable
SHR allows mixed-drive-size RAID
Strong community and documentation

Cons:

Proprietary drive push (HAT5300 drives are expensive)
Base RAM is only 2GB — upgrade needed for Docker
J4125 is aging; no hardware transcoding for 4K HEVC
Price premium over competitors with similar raw specs

2. QNAP TS-264 — Best Features

2-Bay · Intel Celeron N5095 · 8GB DDR4 · ~€350

The QNAP TS-264 is the feature powerhouse. Where Synology focuses on reliability and polish, QNAP throws everything at the wall: HDMI 2.0 output, dual 2.5GbE ports, two M.2 NVMe slots, USB 3.2 Gen 2, and a built-in speaker for alerts. It is a NAS that tries to be a mini server.

The N5095 CPU is faster than Synology's J4125, and the 8GB base RAM means you can run Docker containers, VMs via Virtualization Station, and QNAP's own apps without immediately needing an upgrade. The dual 2.5GbE ports support link aggregation, which is useful if your network backbone supports it.

QNAP's QTS 5 operating system is powerful but less intuitive than DSM. The interface is busier, the settings are scattered across more menus, and the default security posture is weaker — QNAP has had several high-profile vulnerabilities in recent years (Deadbolt ransomware was particularly painful for the community). You need to harden a QNAP NAS on day one: disable default admin, enable 2FA, and configure the firewall.

The real advantage of the TS-264 is HDMI output. If you want a NAS that doubles as a media player connected directly to a TV, this is the only option on the list that does it natively without a separate NUC or Apple TV.

Pros:

Most features per euro (HDMI, dual 2.5GbE, dual M.2)
8GB RAM base — no immediate upgrade needed
Virtualization Station for lightweight VMs
HDMI 2.0 for direct media playback
PCIe slot for 10GbE expansion

Cons:

QTS software is less polished than DSM
Security track record (Deadbolt, Qlocker incidents)
Higher idle power than Synology (~15-18W without drives)
Fan noise higher than Synology at comparable load
Support documentation less comprehensive

3. Terramaster F2-223 — Best Budget

2-Bay · Intel Celeron N4505 · 4GB DDR4 · ~€200

The Terramaster F2-223 is the budget pick that does not feel like a compromise in core functionality. At €200, it undercuts the Synology DS224+ by €100 while offering similar hardware: a dual-core Intel CPU, 4GB RAM (upgradeable), and dual GbE ports.

Terramaster's TOS 5 operating system is the weak point. It has improved significantly from TOS 4, which was genuinely bad, but it still lacks the polish and reliability of DSM or QTS. The Docker integration works, the backup applications exist, and the file sharing protocols (SMB, NFS, WebDAV) are functional — but the overall experience feels like using a first-generation product. The community is smaller, the documentation is thinner, and finding solutions to edge-case problems takes more effort.

Where the F2-223 wins is raw value. The N4505 is a modern dual-core chip that handles file serving and light Docker workloads without issue. The 4GB base RAM is adequate for basic use. The dual GbE ports support link aggregation. And at €200, you can buy two of them for the price of one Synology DS224+ — which, if you need redundancy, is a compelling argument.

Pros:

Best price-to-hardware ratio
4GB RAM base (more than Synology's 2GB)
Dual GbE with link aggregation
Compact, quiet design
Adequate for basic file backup and Docker

Cons:

TOS 5 is significantly less polished than DSM
Smaller community and thinner documentation
No Hyper Backup equivalent — more manual backup setup
Build quality noticeably cheaper than Synology/QNAP
Long-term software support uncertain

4. Synology DS423+ — Best for Growth

4-Bay · Intel Celeron J4125 · 2GB DDR4 (expandable to 32GB) · ~€500

The DS423+ is the NAS you buy when you know your storage needs will grow. Four bays with Synology Hybrid RAID (SHR) means you can start with two drives, add a third later, and a fourth later — without rebuilding the array from scratch each time.

The J4125 CPU is the same as the DS224+, which means the same performance limitations apply. For file serving and backup, it is fine. For media transcoding, it is not. The value of the DS423+ is not in its CPU — it is in the four bays, the expandability, and the Synology ecosystem at a scale that grows with your data.

Max RAM is 32GB (via a single SO-DIMM slot), which is overkill for most home use but enables running Synology's Virtual Machine Manager with multiple lightweight VMs. The two built-in NVMe M.2 slots (for cache, not storage) improve random I/O for database-heavy applications.

Power consumption scales with drives: idle around 15-20W with four drives spun down, 35-45W under load with four active drives. This is the price of four bays — plan your power budget accordingly.

Pros:

SHR with 4 bays — maximum flexibility for mixed drives
Expandable to 32GB RAM
NVMe cache slots for performance-sensitive workloads
Same reliable DSM 7.2 as the DS224+
10GbE expandable via PCIe slot

Cons:

€500 is a significant investment for a 4-bay consumer NAS
J4125 CPU is aging — same as the cheaper DS224+
2GB base RAM requires immediate upgrade
Power consumption with 4 active drives is meaningful
No hardware transcoding

5. UGREEN DXP2800 — Best Newcomer

2-Bay · Intel N100 · 8GB DDR5 · ~€280

UGREEN is the new entrant trying to disrupt the Synology-QNAP duopoly. The DXP2800 is their flagship 2-bay, and on paper, it is impressive: Intel N100 CPU (the same chip powering the best mini PCs), 8GB DDR5 RAM, dual 2.5GbE, two M.2 NVMe slots, and a metal chassis that feels more premium than the price suggests.

UGREEN's UGOS operating system is the wildcard. It is new, still in active development, and lacks the maturity of DSM or QTS. The core features work — file sharing, Docker, basic backup — but the software ecosystem is thin, the app store is limited, and the long-term support commitment is unproven. UGREEN is a hardware company, not a software company, and it shows.

The hardware is genuinely excellent for the price. The N100 is more powerful than Synology's J4125, the 8GB DDR5 RAM eliminates the immediate upgrade need, and the dual 2.5GbE ports are faster than anything Synology offers in this price range. If UGOS matures, the DXP2800 could be the best value NAS on the market. If it does not, you are left with good hardware running mediocre software.

The N100's idle power draw (~8-10W for the enclosure) is competitive with Synology. The DDR5 memory and efficient Intel chip keep thermals low even under sustained file operations.

Pros:

Best hardware-per-euro in the 2-bay category
Intel N100 — more powerful than any Synology in this range
8GB DDR5 base RAM
Dual 2.5GbE and dual M.2 NVMe
Metal chassis, solid build quality

Cons:

UGOS is immature — limited app ecosystem
Long-term software support uncertain (new company in NAS space)
Backup software not comparable to Hyper Backup
Community is small; troubleshooting requires more self-reliance
Firmware update history is short — no proven track record

Head-to-Head Comparison

Feature	DS224+	TS-264	F2-223	DS423+	DXP2800
CPU	J4125 (4C)	N5095 (4C)	N4505 (2C)	J4125 (4C)	N100 (4C)
Base RAM	2GB	8GB	4GB	2GB	8GB
Max RAM	6GB	8GB	32GB	32GB	16GB
Bays	2	2	2	4	2
NIC	2x 1GbE	2x 2.5GbE	2x 1GbE	2x 1GbE	2x 2.5GbE
M.2 Slots	2 (cache)	2 (storage)	0	2 (cache)	2 (storage)
HDMI	No	Yes (2.0)	No	No	No
PCIe Slot	No	Yes	No	Yes	No
Idle Power	~10-12W	~15-18W	~8-10W	~15-20W	~8-10W
OS	DSM 7.2	QTS 5	TOS 5	DSM 7.2	UGOS
Price	~€300	~€350	~€200	~€500	~€280

Buyer's Guide

Buy the Synology DS224+ if: You want a NAS that works reliably for 5+ years with minimal maintenance. DSM is the best NAS OS, Hyper Backup handles every backup scenario, and the ecosystem is proven. This is the default recommendation for 90% of home users.

Buy the QNAP TS-264 if: You need maximum features in a 2-bay form factor. HDMI output for direct media playback, dual 2.5GbE for fast network transfers, and 8GB RAM for Docker-heavy use cases. Be prepared to harden the security on day one.

Buy the Terramaster F2-223 if: Budget is the primary constraint and you are comfortable with a less polished software experience. The hardware is solid for the price, and TOS 5 is functional even if it is not inspiring.

Buy the Synology DS423+ if: You know your storage needs will grow beyond two drives, or you want the option to run VMs alongside file serving. The four bays with SHR give you the most flexible RAID configuration available in a consumer NAS.

Buy the UGREEN DXP2800 if: You are betting on UGOS maturing and want the best hardware value today. The N100 + 8GB DDR5 + dual 2.5GbE combination is unbeatable at €280. Just be aware you are an early adopter.

For the homelab audience: The Synology DS224+ with two 4TB drives in SHR-1 (mirroring) is the backbone of a 3-2-1 backup strategy. Pair it with a cloud sync target (Backblaze B2 or your own offsite S3-compatible endpoint) and you have a backup architecture that survives fire, theft, and ransomware. I run exactly this — the details are in the disaster recovery runbook.

Frequently Asked Questions

How much storage do I need for home backup?

For most households, 2-4TB covers documents, photos, phone backups, and critical system images. If you are archiving video or running a media server, 8-16TB becomes necessary. Start with two drives in a mirrored RAID (RAID 1 or SHR-1), and expand later — the point of a NAS is that you can replace drives one at a time with larger ones.

What drives should I put in a NAS?

NAS-rated drives (Seagate IronWolf, WD Red Plus) are designed for 24/7 operation and vibration tolerance in multi-bay enclosures. Desktop drives (WD Blue, Seagate Barracuda) are cheaper but not rated for continuous use — they work, but the failure rate in a NAS environment is measurably higher. For the home user who wants set-and-forget reliability, buy NAS-rated drives and sleep soundly.

Is RAID the same as backup?

No. RAID protects against drive failure, not against file deletion, corruption, ransomware, or fire. A NAS in RAID 1 with no offsite backup is one rm -rf / away from losing everything. RAID is availability; backup is recoverability. You need both.

Should I use Synology Hybrid RAID or standard RAID?

SHR if you are on Synology — it is strictly better than standard RAID for home use. SHR allows mixing different drive sizes without wasting capacity, and it degrades more gracefully than traditional RAID when a drive fails. For two drives, SHR-1 is identical to RAID 1. For four drives, SHR-1 gives you single-drive fault tolerance with the flexibility to add larger drives later without rebuilding the entire array.

Can I run Docker on these NAS devices?

Yes, all five support Docker. Synology's Container Manager (formerly Docker package) is the most polished implementation. QNAP's Container Station works well but has a more complex interface. Terramaster and UGREEN's Docker support is functional but less integrated. If Docker is a primary use case, the QNAP TS-264 (8GB RAM, N5095 CPU) or UGREEN DXP2800 (8GB RAM, N100 CPU) give you the best hardware foundation.

How long does a NAS last?

A quality NAS (Synology, QNAP) typically lasts 5-8 years before the hardware becomes obsolete or the software support ends. The drives inside have a rated lifespan of 3-5 years for NAS-rated models. Budget for drive replacement every 3-4 years as a preventive measure — drives do not give reliable warning before failing. The NAS enclosure itself will outlast two or three sets of drives.

Is a NAS worth it versus just using an external hard drive?

For a single computer with non-critical data, an external drive is fine. For any multi-device household, any data you cannot afford to lose, or any homelab running services that need shared storage — a NAS is worth the investment. The difference is automation: a NAS handles RAID, snapshots, cloud sync, and multi-user access automatically. An external drive requires you to remember to plug it in and copy files, which is a backup strategy that fails the moment you forget.

Conclusion

The NAS market in 2026 is defined by one question: do you trust the software or the hardware more? Synology's DS224+ has aging hardware but bulletproof software. The UGREEN DXP2800 has excellent hardware but unproven software. The QNAP TS-264 sits in between, with strong hardware and software that requires active security management.

For a home backup NAS that runs 24/7 for the next five years, trust matters more than specs. The Synology DS224+ is the boring, reliable choice — and that is exactly what you want from the device holding your only copy of family photos and critical documents.

Buy the DS224+. Put two IronWolf drives in it. Configure Hyper Backup to sync to a cloud target. Verify the restore works once. Then forget about it — which is exactly how a backup system should behave.

🚀 Azure Zero-Trust Starter Pack — €99

Includes: Acmebot Enterprise VNet, Hub & Spoke Zero-Trust, Azure Firewall Forced Tunneling
Save €48 vs. buying the three modules separately
Bonus: ISO 27001 Auditor Checklist (PDF) - Annex A control map for all 3 modules
Everything to pass your first Azure compliance audit
Full source code for all 3 modules - no lock-in, no black box

Get the enterprise module →

Full source code · one-time payment · instant download

Best Mini PC for Homelab 2026

david — Sun, 19 Jul 2026 10:52:44 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

View the complete homelab infrastructure source on GitHub 🐙

After cycling through more mini PCs than I care to admit in my homelab — some running 24/7, others retired after a single disappointing week — I have settled on a clear ranking for 2026. The mini PC market has matured significantly: Intel's N100 still dominates the efficiency tier, AMD's Ryzen mobile chips crush multi-threaded workloads, and even the budget options no longer feel like compromise.

This comparison is based on real deployment experience. My lab runs Proxmox, K3s, Terraform-managed VMs, and local LLM inference. Every machine on this list has been tested under those workloads.

TL;DR — The Picks

Model	CPU	RAM	Price (approx.)	Best For
BMAX B1 Pro*	Intel N100	16GB	~€180	Best Overall
Beelink SER5*	Ryzen 5 5560U	16GB	~€230	Best Performance
GMKTec NucBox K6*	Ryzen 7 6800H	32GB	~€350	Best for Heavy Workloads
Intel NUC 13 Pro*	i5-1340P	16GB	~€400	Best Build Quality
Trigkey Speed S5*	Intel N100	16GB	~€150	Best Budget

If you want the short version: the BMAX B1 Pro is the best all-rounder for a homelab that needs to run Proxmox, containers, or lightweight VMs 24/7. If you need more compute for local LLM inference or heavy compilation, the GMKTec NucBox K6 with its 6800H and 32GB RAM is worth the premium.

Selection Criteria — What Matters in a Homelab Mini PC

Buying a mini PC for a homelab is fundamentally different from buying one as a desktop replacement. The priorities shift entirely:

Idle power draw — This machine runs 24/7. A 5W difference in idle consumption is €40+ per year. Intel N100 platforms consistently win here, idling at 6-10W versus 15-25W for Ryzen alternatives.

Thermal management under sustained load — Mini PCs live in racks or shelves with limited airflow. A chip that throttles after 20 minutes of sustained compilation is useless for building container images or running batch jobs.

RAM expandability — 16GB is the minimum for a serious Proxmox host running multiple VMs. Prefer models that accept SO-DIMM upgrades to 32GB or 64GB, not soldered memory.

Network interfaces — A single 1GbE port is a bottleneck for any hypervisor. Dual NICs or at least a 2.5GbE port makes the machine viable as a Proxmox node without USB Ethernet dongles.

Storage options — M.2 NVMe is table stakes. A second drive bay (2.5" SATA or a second M.2 slot) for Proxmox local storage or TrueNAS is a major advantage.

Noise — If this thing sits in your office, fan noise at load matters. Some N100 boxes are fanless; most Ryzen units need active cooling under load.

Individual Reviews

1. BMAX B1 Pro — Best Overall

Intel N100 · 16GB DDR5 · 500GB NVMe · ~€180

The BMAX B1 Pro hits the sweet spot for a homelab node. The Intel N100 is a 6W TDP chip that idles around 7-8W wall draw, making it nearly invisible on your electricity bill when running 24/7. It has a single M.2 NVMe slot and a 2.5" SATA bay, giving you enough storage for Proxmox OS plus a reasonable local datastore.

Where it really shines is reliability. I have had BMAX units running in my lab for over a year with zero hardware failures. The DDR5 memory is a nice upgrade from the DDR4 N100 boxes — not because N100 needs the bandwidth, but because DDR5 SO-DIMMs are easier to source at higher capacities for future upgrades.

The 2.5GbE Realtek NIC is the one weak point. Realtek drivers on Linux work, but they lack the kernel support depth of Intel i225/i226 NICs. For a Proxmox node running Terraform-managed VMs, it works fine. For serious packet inspection or high-throughput bridging, you will notice the difference.

Pros:

Excellent idle efficiency (7-8W)
DDR5 + SATA expansion bay
Compact, quiet under light load
Proven reliability over 12+ months

Cons:

Realtek 2.5GbE NIC (not Intel)
Only 1x M.2 slot + 1x SATA
Single-channel memory

2. Beelink SER5 — Best Performance

Ryzen 5 5560U · 16GB DDR4 · 500GB NVMe · ~€230

The Beelink SER5 is the performance-per-euro champion. The Ryzen 5 5560U offers 6 cores / 12 threads with a 15W TDP, delivering roughly 2.5x the multi-threaded performance of the N100 while only costing €50 more. For containerized workloads — building Docker images, running Kubernetes operators, or hosting development databases — the extra cores translate directly into faster builds and lower latency.

The tradeoff is power. Idle draw sits around 18-22W, which over a year adds up. If this machine is your primary Proxmox host running 10+ containers, the performance justifies the extra power. If it is just running Pi-hole and a couple of lightweight services, the N100 is the smarter buy.

The SER5 has a single M.2 NVMe slot and a 2.5" SATA bay, plus dual USB 3.2 ports for external storage. The dual 2.5GbE ports (Realtek on some revisions, Intel on newer batches — check before buying) make it viable as a multi-VLAN Proxmox trunk node.

Pros:

Strong multi-threaded performance for the price
Dual 2.5GbE (Intel NIC on newer revisions)
Upgradeable to 64GB DDR4
Good build quality for the price tier

Cons:

Higher idle power (18-22W)
Fan audible under sustained load
DDR4 not DDR5 (less future-proof)

3. GMKTec NucBox K6 — Best for Heavy Workloads

Ryzen 7 6800H · 32GB DDR5 · 1TB NVMe · ~€350

If you need raw compute in a small box, the GMKTec K6 is the answer. The 6800H is a 45W class chip — significantly more powerful than anything else on this list — and it comes with 32GB of DDR5 RAM out of the box. That is enough to run Proxmox with several VMs, a K3s cluster, and still have headroom for local LLM inference with Ollama.

The power draw reflects the performance: idle around 20-28W, and under full load it can hit 55-65W. This is not a machine you deploy for Pi-hole. This is a machine you deploy when you need one box to replace three lesser ones.

The cooling system on the K6 is beefier than the BMAX or Beelink — dual copper heat pipes and a larger fan. Under sustained compilation loads (building a K3s cluster from source, for example), temperatures stay below 80°C, which is impressive for this form factor. The fan is audible but not obnoxious; I would not put it on a desk next to my ear, but in a rack shelf it disappears.

Storage is generous: 2x M.2 NVMe slots (one PCIe 4.0, one PCIe 3.0) plus a 2.5" SATA bay. This makes it viable as a small Proxmox + local storage server without needing a separate NAS for warm data.

Pros:

Serious multi-thread performance (8 cores / 16 threads)
32GB DDR5 included
Dual M.2 NVMe + SATA
Good thermal design for sustained loads

Cons:

High idle power for a mini PC (20-28W)
Larger and heavier than competitors
Fan noise at load — not suitable for a quiet office shelf
Price premium over N100/Ryzen 5 alternatives

4. Intel NUC 13 Pro — Best Build Quality

Intel Core i5-1340P · 16GB DDR5 · 500GB NVMe · ~€400

The Intel NUC 13 Pro is the premium choice, and the price reflects it. Intel's own NUC line (now maintained by ASUS after the brand transfer) has the best build quality in this category: the chassis is solid aluminum, the thermal design is refined, and the component selection — particularly the Intel i226-V 2.5GbE NIC — is the best you will find.

The i5-1340P is a 12-core / 16-thread chip that sits between the Ryzen 5 5560U and the 6800H in performance. Intel's hybrid architecture (P-cores + E-cores) works well for heterogeneous workloads: the P-cores handle bursty compilation tasks while the E-cores absorb background services without burning power.

The i226-V NIC deserves special mention. Intel's 2.5GbE controllers have the best Linux driver support of any vendor. If you are running Proxmox with VLAN bridging, OVS (Open vSwitch), or any kind of advanced networking, the Intel NIC eliminates a whole category of troubleshooting. I have spent enough time debugging Realtek driver issues on N100 boxes that this alone justifies the premium for a production homelab node.

The downside is pure cost. At €400, you are paying more than double the BMAX B1 Pro for a machine that, in raw compute benchmarks, is only marginally faster than the Beelink SER5 at €230. You are paying for build quality, the Intel NIC, and NUC reliability — which matters when this machine runs your production K3s cluster.

Pros:

Best-in-class build quality and thermal design
Intel i226-V 2.5GbE NIC (best Linux driver support)
VPro/AMT for remote out-of-band management
Excellent idle efficiency for a 12-core chip (~12-15W)

Cons:

Significant price premium
Only 1x M.2 + 1x 2.5" SATA
Performance gap vs. cheaper options is marginal
ASUS NUC availability can be inconsistent in EU

5. Trigkey Speed S5 — Best Budget

Intel N100 · 16GB DDR4 · 500GB NVMe · ~€150

The Trigkey Speed S5 is the BMAX B1 Pro's budget cousin. Same Intel N100 chip, same 16GB RAM, same basic form factor — but at €150 it undercuts the BMAX by €30. For a first homelab node or a dedicated Pi-hole/AdGuard Home box, this is the price-to-performance king.

The tradeoffs are predictable: the chassis feels cheaper (more plastic, less metal), the thermal solution is more basic (the fan ramps up more aggressively under load), and the expansion options are limited — single M.2 NVMe only, no SATA bay. The NIC is Realtek 2.5GbE, same as the BMAX.

For a dedicated role — running a single service like DNS, a reverse proxy, or a lightweight monitoring stack — these compromises are irrelevant. The N100 chip is identical, and the performance is identical. You just get a slightly noisier box with less upgrade potential.

Pros:

Lowest price for N100 + 16GB platform
Same N100 performance as more expensive options
Adequate for single-purpose homelab roles

Cons:

No SATA expansion bay
Cheaper chassis and thermal solution
Fan more aggressive under load
Limited brand support/longevity compared to BMAX or Beelink

Head-to-Head Comparison

Feature	BMAX B1 Pro	Beelink SER5	GMKTec K6	Intel NUC 13	Trigkey S5
CPU	N100 (4C/4T)	5560U (6C/12T)	6800H (8C/16T)	i5-1340P (12C/16T)	N100 (4C/4T)
RAM	16GB DDR5	16GB DDR4	32GB DDR5	16GB DDR5	16GB DDR4
Max RAM	32GB	64GB	64GB	64GB	32GB
M.2 Slots	1	1	2	1	1
SATA Bay	Yes	Yes	Yes	Yes	No
NIC	1x 2.5GbE (RTL)	2x 2.5GbE	2x 2.5GbE (Intel)	1x 2.5GbE (Intel i226)	1x 2.5GbE (RTL)
Idle Power	~7-8W	~18-22W	~20-28W	~12-15W	~7-8W
Fanless Option	No (very quiet)	No	No	No	No
Price	~€180	~€230	~€350	~€400	~€150

Buyer's Guide

Buy the BMAX B1 Pro if: You want a reliable, efficient all-rounder that sips power and handles Proxmox, containers, and lightweight VMs without fuss. This is the default recommendation.

Buy the Beelink SER5 if: You need more CPU power than the N100 but want to keep costs under €250. Ideal for a primary Proxmox host running a mix of containerized and VM workloads.

Buy the GMKTec K6 if: You want one box to rule them all. The 32GB RAM and 6800H handle local LLM inference, heavy compilation, and multi-VM Proxmox without breaking a sweat. Accept the higher power draw.

Buy the Intel NUC 13 Pro if: Build quality and NIC reliability matter more than price. The i226-V NIC and VPro/AMT remote management make this the "set and forget" choice for a production homelab that you do not want to babysit.

Buy the Trigkey S5 if: Budget is the primary constraint, or you need a dedicated single-purpose node (DNS, monitoring, reverse proxy) where the N100 is more than enough.

If you are starting from scratch: Buy two BMAX B1 Pro units. Run Proxmox with a VM or container failover strategy. The total cost (€360) is less than a single Intel NUC, and you get hardware redundancy that no single mini PC can provide regardless of its specs.

Frequently Asked Questions

How much RAM do I need for a homelab mini PC?

16GB is the practical minimum if you are running Proxmox with more than one VM or container stack. For K3s with a few worker pods and monitoring (Prometheus, Grafana), 32GB gives you breathing room. 64GB only makes sense if you are running local LLM inference or multiple heavy databases. The N100 platforms max out at 32GB; Ryzen and Intel i5 platforms support 64GB.

Should I care about DDR4 vs DDR5 in a mini PC?

For N100 chips, the difference is negligible — the N100's memory bandwidth ceiling is low enough that DDR4 and DDR5 perform identically in practice. DDR5 matters on the Ryzen 7 and Intel i5 platforms where the memory controller can actually use the extra bandwidth. The real DDR5 advantage is availability: DDR5 SO-DIMMs are easier to find at 32GB and 48GB capacities now, making future upgrades simpler.

Can a mini PC replace a full-size server for homelab use?

For 80% of homelab use cases, yes. A single Ryzen 7 mini PC with 32GB RAM can run Proxmox with 5-8 VMs, a K3s cluster, monitoring, and local storage — enough for most self-hosting setups. What it cannot replace: high-storage NAS (you still need separate disks), GPU-accelerated workloads (no discrete GPU), and high-availability clusters (you need at least 2-3 nodes for proper failover).

What about noise levels?

N100 boxes (BMAX, Trigkey) are nearly silent at idle and quiet under moderate load — you will not hear them from a meter away. The Beelink SER5 and GMKTec K6 have audible fans under sustained load; the K6 is the loudest of the group. If noise is a primary concern, the BMAX B1 Pro or Intel NUC 13 Pro are the quietest options with meaningful compute.

Should I buy one powerful mini PC or multiple cheaper ones?

Multiple cheaper ones, almost always. A single point of failure is a single point of failure, regardless of how powerful it is. Two BMAX B1 Pro units (€360 total) give you hardware redundancy and the ability to migrate VMs between hosts during maintenance. The only exception: if you need GPU compute for local LLMs, in which case a single GMKTec K6 with its Radeon 680M is the minimum viable platform.

Do these mini PCs support Proxmox well?

All of them work with Proxmox. The Intel NUC 13 Pro has the smoothest experience thanks to the Intel i226-V NIC and full kernel support. The N100 boxes work fine with Proxmox but may need the r8169 driver tuned for the Realtek NIC under heavy bridged traffic. Ryzen platforms (Beelink, GMKTec) work well out of the box. I have personally run Proxmox on all five of these platforms without blocking issues.

What about the Raspberry Pi 5 as a homelab node?

The Raspberry Pi 5 is a legitimate homelab option — I run two in my own lab for edge DNS and Keepalived — but it does not compete with mini PCs for compute-intensive workloads. The Pi 5's ARM Cortex-A76 cores are roughly 40% slower than an N100 in single-threaded work and lack the x86 instruction set compatibility that Proxmox and many Docker images assume. For lightweight services, the Pi 5 wins on power efficiency (idle ~3-4W). For anything beyond DNS, reverse proxy, and monitoring, a mini PC is the better investment.

Conclusion

The mini PC market for homelab use in 2026 is excellent. There has never been a better time to buy: even the €150 budget option gives you enough compute to run a serious self-hosted stack. The choice comes down to your workload profile and how much you value power efficiency versus raw performance.

For my money, the BMAX B1 Pro at €180 is the default answer. It runs 24/7, sips power, and handles everything I throw at it. But if you are building a serious lab with local AI, multiple VM clusters, and heavy CI workloads, the GMKTec K6 at €350 is the machine that grows with you.

Whatever you pick: buy two. Redundancy beats raw performance every time in a homelab.

🚀 Azure Zero-Trust Starter Pack — €99

Includes: Acmebot Enterprise VNet, Hub & Spoke Zero-Trust, Azure Firewall Forced Tunneling
Save €48 vs. buying the three modules separately
Bonus: ISO 27001 Auditor Checklist (PDF) - Annex A control map for all 3 modules
Everything to pass your first Azure compliance audit
Full source code for all 3 modules - no lock-in, no black box

Get the enterprise module →

Full source code · one-time payment · instant download

Zero NetworkPolicies on Vault: How I Found the Biggest Gap in My Cluster and a GitOps Tracking Bug That Hid It

david — Sun, 19 Jul 2026 10:43:16 +0000

Disclosure: This post contains Amazon affiliate links (marked with *). If you buy through them, I earn a small commission at no extra cost to you. I only link gear I actually own and use daily.

A cross-cutting isolation check across every namespace in my k3s cluster revealed that Vault — the trust root every ExternalSecret reads from — had zero NetworkPolicies. Any pod in any namespace, including a compromised Immich or a misconfigured application, could reach Vault's API on port 8200 with no network-layer restriction.

The fix was straightforward. The investigation that followed uncovered a second, more insidious problem: a class of ArgoCD drift where merged git changes silently do nothing.

The Finding

kubectl get networkpolicy -n vault
# No resources found in vault namespace

Every other sensitive namespace had at least a basic ingress policy. Vault had nothing. The trust root — the service that every ExternalSecret in the cluster resolves secrets from — was wide open at the network layer.

Why This Matters

The blast radius of a Vault compromise is the entire cluster. Every ExternalSecret references a ClusterSecretStore pointing at Vault. If Vault is compromised, every secret the cluster trusts is compromised. Network segmentation is the first line of defense.

Mapping the Real Traffic Pattern

Before writing any policy, I mapped who actually talks to Vault:

# Who's calling Vault right now?
kubectl exec -n vault vault-0 -- \
  cat /vault/logs/audit-backend.file-c-audit.log | \
  jq -r '.request.connection.remoteAddress' | \
  sort | uniq -c | sort -rn

The result was simpler than expected:

external-secrets namespace — the External Secrets Operator's central controller calls Vault via the ClusterSecretStore
vault namespace — vault-unseal and vault-0 (same-namespace communication)

No other pod in the cluster calls Vault directly. ESO resolves centrally and writes plain Kubernetes Secrets into each app's namespace. So the policy only needs to allow cross-namespace access from external-secrets — not from every app namespace.

The Policy

# kubernetes/system/vault/network-policies.yml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: vault-allow-external-secrets
  namespace: vault
spec:
  podSelector: {}  # all pods in vault namespace
  policyTypes:
    - Ingress
  ingress:
    # Allow External Secrets Operator to reach Vault API
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: external-secrets
      ports:
        - port: 8200
          protocol: TCP
    # Allow same-namespace (vault-unseal, vault-0)
    - from:
        - podSelector: {}

The key insight: external-secrets is the only cross-namespace caller. Individual app pods never touch Vault — ESO handles it centrally. This means the policy is narrow by design, not by guessing.

Live-Tested Before Committing

This isn't a "write and hope" policy. I tested it live with selfHeal temporarily disabled:

# 1. Delete a live ExternalSecret's backing Secret
kubectl delete secret renovate-token -n external-secrets

# 2. Verify ESO recreates it through the new policy
kubectl get secret renovate-token -n external-secrets
# → recreated within 5s — proves ESO→Vault traffic works

# 3. Verify a pod OUTSIDE the allowed namespaces is blocked
kubectl run test-pod --rm -it --image=busybox -n apps -- \
  wget -q -O- --timeout=3 http://vault.vault.svc:8200/v1/sys/health
# → timeout — proves the policy actually restricts

# 4. Restore selfHeal

The policy works in both directions: legitimate traffic flows, unauthorized traffic is blocked.

The Second Problem: ArgoCD Tracking Gap

While investigating the Vault namespace, I found that kubernetes/apps/network-policies.yml and traefik-websockets.yml were loose files directly under kubernetes/apps/ — not in subdirectories.

The homelab-apps ApplicationSet generates Applications per subdirectory:

# homelab-apps ApplicationSet
spec:
  generators:
    - directories:
        - path: kubernetes/apps/*

Loose files under kubernetes/apps/ are never picked up. They're not subdirectories. No Application watches them.

# Check if ArgoCD tracks these files
kubectl get applications -A -o json | \
  jq -r '.items[] | select(.spec.source.path | startswith("kubernetes/apps/")) | .spec.source.path'
# → kubernetes/apps/*/ (only subdirectories)

The Drift

These files had kubectl.kubernetes.io/last-applied-configuration annotations — they were applied by hand at some point, never reconciled against git since. Real drift had already happened:

network-policies.yml still excluded Atlantis from allow-intra-namespace even though Atlantis moved to its own LXC months ago
A stale allow-atlantis-ingress policy matched zero live pods — dead code, never pruned

The Fix: Self-Watching Application

# kubernetes/apps/gitops-tracking-application.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitops-tracking
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/dwoitzik/homelab-infrastructure.git
    path: kubernetes/apps
    directory:
      recurse: false
      exclude: "*/"
  syncPolicy:
    automated:
      selfHeal: true

This Application watches kubernetes/apps/ itself — including loose files. It's the same pattern as monitoring/manifests-application.yml, but applied to the apps directory.

Important: This file includes itself in its own source path. That's intentional — it's a self-watching bootstrap. The first sync reconciles all previously orphaned files at once.

The Orphaned Bootstrap Class

This isn't just about two files. The sweep found 18 of 41 live Applications with their defining .yml file untracked by any parent Application:

kubernetes/system/*/  →  no ApplicationSet, no recurse generator
kubernetes/apps/*.yml →  ApplicationSet only watches subdirectories

For kubernetes/apps/*, the subdirectory generator covers everything inside directories. But loose files at the root level fall through. For kubernetes/system/*, nothing watches individual Application files at all — each system app is manually kubectl apply'd.

The Pattern

# How to find orphaned Applications in any ArgoCD cluster
for app in $(kubectl get applications -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do
  ns=$(echo $app | cut -d/ -f1)
  name=$(echo $app | cut -d/ -f2)
  path=$(kubectl get application $name -n $ns -o jsonpath='{.spec.source.path}')

  # Check if any generator covers this path
  covers=$(kubectl get applicationsets -A -o json | \
    jq -r ".items[].spec.generators[].directories[].path" | \
    grep -c "$(dirname $path)")

  if [ "$covers" -eq 0 ]; then
    echo "ORPHANED: $ns/$name → $path"
  fi
done

Lessons

1. Network policies should be based on real traffic patterns, not assumptions. I didn't guess which pods talk to Vault — I checked the audit logs. The result was narrower than expected: only external-secrets needs cross-namespace access.

2. Test policies live before committing. Delete a Secret, confirm ESO recreates it. Run a test pod, confirm it's blocked. This catches both over-permissive and over-restrictive policies.

3. ArgoCD ApplicationSets have blind spots. Directory generators don't watch loose files. If you have files directly under the watched path (not in subdirectories), they're orphaned. Either move them into subdirectories or create a tracking Application.

4. Orphaned files drift silently. The git edit happens, gets merged, and everyone assumes ArgoCD picked it up. It didn't. The only way to catch this is to audit which files are actually watched.

Further reading: For understanding the full blast radius of a Vault compromise, Zero Trust Networks by Evan Gilman and Doug Barth* covers the network segmentation principles behind "never trust, always verify." The ArgoCD tracking gap is a common GitOps blind spot — GitOps and Kubernetes by Billy Yuen et al.* covers ApplicationSet generators and their failure modes in depth.

🚀 Azure Firewall - Enterprise Forced Tunneling Edition — €49

Cycle-error-free resource ordering - deploys first time, every time
KMS & Azure AD bypass routes - no broken Windows VMs or auth failures
Dynamic for_each subnet binding - scales to any number of Spokes
IP Group-based firewall policies - no hardcoded IP addresses
FQDN baseline rules for Windows Updates and core Microsoft services

Get the enterprise module →

Full source code · one-time payment · instant download

Vault is the trust root. If its namespace has no NetworkPolicies, you've built zero-trust everywhere except the one place that matters most. And if your GitOps tool isn't watching the files you think it's watching, the "git is the source of truth" guarantee doesn't apply to those files.

Renovate OOMKilled Three Times: Why the Fix Wasn't More Memory

david — Fri, 17 Jul 2026 10:01:45 +0000

Renovate kept getting OOMKilled on the BMAX Mini PC* running my entire k3s cluster. The Kubernetes CronJob would start, run for a while, and then die with exit code 137. The natural assumption: not enough memory. Bump the limit. Problem solved.

Three rounds of bumps later — 2 GiB, then 3 GiB, then 4 GiB — it was still dying. The fix had nothing to do with memory limits. It had to do with what Renovate was actually doing with the memory it had.

View the complete homelab infrastructure source on GitHub 🐙

The Escalation

Round 1: the CronJob was OOMKilled at 2 GiB. Every job, 3 retries each, all dying. kubectl describe pod confirmed exit code 137, reason OOMKilled. Bumped to 3 GiB container limit / 2304Mi V8 heap, keeping the same ~25% non-heap headroom ratio as the previous configuration.

Round 2: OOMKilled again at 3 GiB. Same pattern. Checked for regressions — no commits landed between the last clean run and the first failure that meaningfully grew the dependency count. The internal-docs relocation merged after the failures started, and Renovate doesn't scan .md files anyway. Read as ordinary run-to-run variance tipping an already-marginal limit over, not a new regression.

Round 3: bumped to 4 GiB. This time, it landed at 3 GiB instead of 4 GiB — the edit was described but never actually made before merge. A separate PR corrected it to the intended 3 GiB → 4 GiB / 2304Mi → 3072Mi heap.

Still OOMKilled.

The Diagnostic Run

At this point, guessing at memory limits wasn't working. The approach shifted to isolating the spike instead of outrunning it.

I ran Renovate off-cluster in an isolated diagnostic run with 12 GiB of headroom, a real GitHub token, real branches, and user authorization. The goal: measure the actual peak instead of guessing at it.

The diagnostic run finished clean at exit 0. Peak memory usage: 3.99 GiB.

That confirmed this was never unbounded growth. The 2 GiB and then 4 GiB production kills both landed mid-spike, ~30-40 MiB past the ceiling. The cgroup limit sat too close to a single transient spike for GC to reclaim before the kill.

The Root Cause: Terraform Hash Concurrency

The spike wasn't random. It was Renovate's Terraform manager SHA256-hashing all 14 platform builds of a major hashicorp/aws provider bump (5.x → 6.53.0) to keep .terraform.lock.hcl valid.

At the default concurrentRequestLimit of 16, that's up to 16 provider zips buffered in memory simultaneously — each one a large binary being hashed for lock-file integrity. The spike is predictable: any Terraform provider upgrade that touches multiple platforms hits this.

The fix wasn't more memory. It was flattening the spike at the source:

{
  "terraform": {
    "concurrentRequestLimit": 2
  }
}

Cap concurrent Terraform hash operations at 2 for releases.hashicorp.com. The spike shrinks because fewer provider zips are buffered simultaneously. The 4 GiB limit becomes genuine headroom instead of a near-miss ceiling, and the 6 GiB / 4608Mi heap becomes defense in depth — sized from the actual measured 3.99 GiB peak with real GC margin, not a blind guess.

resources:
  limits:
    memory: 6Gi
  env:
    - name: V8_FLAG_INITIAL_HEAP_SIZE
      value: "4608"

The CronJob stays suspended — this fix doesn't re-enable it. That's a separate decision once the fix is reviewed and validated in production.

The Lesson

The first two bumps were the obvious response to OOMKilled: more memory. They were also wrong, because they treated the symptom (process needs more memory) instead of the cause (process is buffering too many large objects simultaneously).

The diagnostic run with 12 GiB of headroom was the turning point. Instead of guessing at limits, I isolated the real peak. 3.99 GiB. Then the question became: why is the peak 3.99 GiB, and can I flatten it instead of accommodating it?

concurrentRequestLimit=2 flattens it. The memory limit becomes a safety net, not the fix.

This is the same pattern as tuning Kubernetes resource requests: you don't set requests based on what the app uses at peak, you set them based on what it needs to function normally, and you use limits as a ceiling for the spikes. In this case, the "normal" was far below the spike, and the spike was artificially inflated by a configuration knob nobody had touched.

Right-sizing infrastructure isn't just a Kubernetes problem — Azure VM sizing, AKS node pool autoscaling, and Cosmos DB RU allocation all face the same trap: responding to peak usage with more capacity instead of understanding what's driving the peak.

Discord Voice Choppy? It Was Bufferbloat — Fixed with 51 Lines of Terraform

david — Fri, 17 Jul 2026 09:58:12 +0000

Discord voice was choppy. Not for me hearing others — for others hearing me. Robotic, garbled audio that came and went. Confirmed clean over mobile data, which meant the problem was my home network path, not Discord or the ISP.

The investigation led to a textbook bufferbloat scenario on the MikroTik RB5009*: zero QoS or AQM on the WAN interface, a 50 Mbit upload ceiling that's trivially easy to saturate, and RouterOS's default pfifo queue doing nothing to prevent one bulk transfer from monopolizing the link against latency-sensitive UDP voice traffic.

View the complete homelab infrastructure source on GitHub 🐙

The Investigation

First step: check what's actually running on the WAN. /queue simple and /queue tree were both empty. The MikroTik RB5009 was falling back to RouterOS's default pfifo (plain FIFO, no active queue management) on ether1 (WAN).

ISP profile is asymmetric cable: 1000 Mbit down, 50 Mbit up. Real measured value via repeated live speedtests — not taken from MySpeed's configured target, which happened to match but wasn't the authoritative source. A 50 Mbit upload ceiling is easy to saturate, and with no AQM, nothing stops the router's own send buffer from filling and inflating latency for everything sharing the link.

The key question: what's actually saturating the upload? I traced the WAN-upload consumers:

Garage/PBS backups: LAN-local, don't cross WAN. Ruled out.
PBS Google Drive offsite sync: Already disabled (REL-051). Ruled out.
Image pulls: Downloads, not uploads. Ruled out.
Cloudflare-tunneled remote Jellyfin/Immich streaming: The realistic WAN-upload consumer.

That's the traffic profile: mostly latency-insensitive bulk uploads (media streaming through Cloudflare), with Discord's UDP voice flow competing for the same 50 Mbit.

Why Simple Queues, Not Queue Trees

The obvious RouterOS QoS approach is mangle-mark + queue-tree: mark packets by type, apply queue-tree rules to the marks. The problem: fasttrack is enabled globally (see firewall_deterministic.tf), and fasttrack bypasses mangle marking for established connections. A mangle-mark + queue-tree design would silently stop protecting most real traffic — the very traffic that needs protection.

An address-based Simple Queue sidesteps this entirely. It enforces at the interface's queuing layer, independent of fasttrack and marks. The target is a plain IP-range match, no mangle dependency.

The Fix: PCQ Per-Flow Fairness

The actual fix is 51 lines of Terraform, with the comment block doing most of the heavy lifting:

# terraform/stacks/network/qos.tf

resource "routeros_queue_simple" "wan_egress_sqm" {
  name      = "wan-egress-sqm"
  target    = ["10.0.0.0/16"]
  max_limit = "45M/900M"
  queue     = "pcq-upload-default/pcq-download-default"
  comment   = "SQM: cap WAN link under real measured ISP ceiling (1000/50 Mbit) + PCQ per-flow fairness so one bulk transfer can't fill the buffer and starve latency-sensitive traffic (Discord voice bufferbloat fix)"
}

Three details that matter:

1. max_limit = "45M/900M" — 90% of the real ceiling, not 100%. Capping below the ISP's own ceiling keeps this router's queue as the actual bottleneck instead of the ISP modem's. Without that margin, the real bottleneck (and its buffer) sits inside the ISP modem, outside this queue's control, and bufferbloat would still happen there instead.

2. queue = "pcq-upload-default/pcq-download-default" — per-flow fairness. PCQ round-robins per-flow, so one bulk transfer (the Cloudflare-tunneled Jellyfin/Immich streaming) gets its own sub-queue instead of monopolizing the link against Discord's UDP voice flow. This is the standard RouterOS-native SQM approach when CAKE/fq_codel isn't available — I checked this RouterOS build's queue types (/queue type print): only pfifo/bfifo/red/sfq/pcq/mq-pfifo. No CAKE.

3. target = ["10.0.0.0/16"] — all VLANs in one queue. Deliberately not split per-VLAN yet. The goal right now is fixing the shared WAN bottleneck, not per-VLAN prioritization. All five VLANs (10.0.10-100.0/24) share this WAN link.

Why Not mangle + queue-tree

This deserves emphasis because it's the gotcha that would make this fix silently do nothing:

Fasttrack is enabled globally on this router. Fasttrack processes established connections in the fast path, bypassing the mangle chain entirely. If I'd used mangle rules to mark packets and a queue-tree to shape them, the marking would never happen for established connections — which is every real traffic flow after the first packet. The queue-tree would only affect new, short-lived flows, not the sustained bulk transfers that cause bufferbloat.

The Simple Queue approach works because it operates at the interface queuing layer, not the packet marking layer. It doesn't care whether fasttrack is enabled — it applies to all traffic on the target regardless.

Validation

Terraform: fmt -check, validate, tflint all clean. The resource is straightforward — one routeros_queue_simple with a comment that explains the reasoning.

Not applied yet. Opening as a PR for atlantis plan review per my explicit requirement to see the plan before any apply on the network stack. This is the gateway router — a misconfiguration here affects every device in the apartment.

QoS at the network edge — bufferbloat mitigation, per-flow fairness, bandwidth shaping — is the same principle as Azure's traffic shaping with NSGs, Application Gateway rate limiting, and Azure Front Door priority routing. The tools differ, the problem is identical: latency-sensitive traffic competing with bulk transfers on a shared link.

Kubernetes Health Probes: The Host Header Trap That Restarts Healthy Pods

david — Fri, 17 Jul 2026 09:54:04 +0000

On a BMAX Mini PC* running 20+ workloads across a three-node k3s cluster, I had about 20 long-lived services running without a single health probe. A process that hangs — not crashes, but stops responding — would go undetected forever. Kubernetes would keep routing traffic to an unresponsive pod because there's no mechanism to say "this pod is alive but broken."

The fix sounded simple: add livenessProbe and readinessProbe to every deployment. The reality was six batches of probes, each live-tested against the running pods, and a gotcha on batch one that restarted a healthy pod because of a host-header mismatch nobody warned me about.

View the complete homelab infrastructure source on GitHub 🐙

The Host Header Trap

Homepage was batch one. I added a standard httpGet probe against /api/healthcheck on port 3000. Curled it manually from a busybox pod: 200, probe works. Deployed it. Pod immediately entered a restart loop.

kubectl describe pod showed 10 readiness failures and 1 liveness failure. The probe was failing because HOMEPAGE_ALLOWED_HOSTS is set to home.woitzik.dev plus loopback — a security setting that prevents host-header-based attacks. When kubelet sends an httpGet probe, it sends the Pod IP as the Host header. Homepage's Next.js host-validation middleware sees 10.0.20.147 in the Host header, it's not in the allowlist, it returns 400. Kubelet treats 400 as failure.

The fix is a Host header override on the probe, not loosening the allowlist:

livenessProbe:
  httpGet:
    path: /api/healthcheck
    port: 3000
    httpHeaders:
      - name: Host
        value: localhost
readinessProbe:
  httpGet:
    path: /api/healthcheck
    port: 3000
    httpHeaders:
      - name: Host
        value: localhost

The same issue hit Nextcloud (NEXTCLOUD_TRUSTED_DOMAINS only permits nextcloud.woitzik.dev) and Paperless (PAPERLESS_ALLOWED_HOSTS is an explicit domain/IP allowlist). All three needed the Host: localhost override. Without it, the probes fail on a security feature doing exactly what it's supposed to do.

If your app has any host-header validation, your kubelet probes will fail without this override. This is the single most common gotcha in health probe configuration, and almost no tutorial mentions it.

Batch 1: Authelia, Garage, Vaultwarden

Started with the three highest-blast-radius services where the probe endpoint could be confirmed with high confidence:

Authelia: /api/health on port 9091, returns {"status":"OK"}. Already the exact endpoint blackbox-exporter scrapes externally, so this was a known-good path, not a guess.
Garage: TCP-only on the S3 port. No wget or curl in the image to verify an HTTP health path against live first. TCP probe it is.
Vaultwarden: Same — TCP-only, no HTTP health endpoint available in the image.

# Authelia — HTTP probe, known-good endpoint
readinessProbe:
  httpGet:
    path: /api/health
    port: 9091
  initialDelaySeconds: 5
  periodSeconds: 10

# Garage — TCP probe, no HTTP healthz in the image
readinessProbe:
  tcpSocket:
    port: 2335
  initialDelaySeconds: 5
  periodSeconds: 10

# Vaultwarden — TCP probe, same reasoning
readinessProbe:
  tcpSocket:
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 15

All three were live-tested by temporarily pausing ArgoCD's selfHeal, applying the probes, confirming READY 1/1 with zero restarts, then restoring selfHeal. Not assumed, not deployed blind.

Batch 2: Cloudflared, Gitea, Headscale, Home Assistant, Homepage

Each endpoint was tested before writing the probe via port-forward or in-container curl against the running pod:

cloudflared: /ready on its metrics port (20241). Reports actual tunnel connection count — catches a process that's alive but has lost its tunnel to Cloudflare's edge. Required adding --metrics 0.0.0.0:20241 so kubelet can reach it from outside the pod network namespace.
gitea: /api/healthz → {"status":"pass"}
headscale: /health on the HTTP port → {"status":"pass"}
home-assistant: /manifest.json — public, no auth token needed unlike /api/*, returns 200 once the frontend is serving
homepage: /api/healthcheck → "up" (with the Host: localhost override from the gotcha above)

Generous initialDelaySeconds and failureThreshold on each to avoid flapping during normal startup. Home Assistant especially can take 30+ seconds to fully initialize its integrations.

Batch 3: Keel, Mealie, MySpeed, SearXNG, Uptime Kuma

All tested via Pod IP using a temporary busybox pod or a same-namespace pod's wget — not localhost:

keel: /healthz → 200, empty body
mealie: /api/app/about → public JSON, no auth needed
myspeed: / → 200 frontend HTML
searxng: / → 200 search page HTML, no host-validation issue
uptime-kuma: / → 302 redirect to /dashboard then 200. Kubelet's httpGet probe treats any 200 <= code < 400 as success, so the redirect itself passes.

The lesson from batch one: always test probes via Pod IP, not localhost. A probe that works with curl localhost:3000 inside the container might fail when kubelet hits the Pod IP, because the app sees a different Host header.

Batch 4: Immich Stack

Immich had zero probes across all four workloads — postgres, valkey, server, and ml. One logical stack, so all four fixed together:

immich-postgres: pg_isready -U immich → accepting connections
immich-valkey: valkey-cli ping → PONG
immich-server: /api/server/ping → {"res":"pong"}, no auth required
immich-ml: /ping → pong. Longer initialDelaySeconds since CLIP and face-detection models load into memory on startup before this endpoint responds.

Batch 5: Nextcloud, Paperless, Open-WebUI

The final batch, same Host header gotcha as Homepage:

postgres-nextcloud: pg_isready → accepting connections
redis-nextcloud: redis-cli ping → PONG
nextcloud: /status.php, Host: nextcloud.woitzik.dev override (NEXTCLOUD_TRUSTED_DOMAINS only permits that domain)
postgres-paperless: pg_isready → accepting connections
redis-paperless: redis-cli ping → PONG
paperless: /api/, Host: docs.woitzik.dev override. Returns 302 to schema/view/ which is a valid pass (200 <= code < 400).
paperless-gotenberg: /health → real component-status JSON
paperless-tika: / → 200 index page (no lighter healthz in this version)
open-webui: /health → {"status":true}
paperless-gpt: / → 200 frontend HTML. No k8s Service exists for this one, but probes work via Pod IP regardless.

The Pattern

Every probe was live-tested against the actual running pod before committing. Not guessed from docs, not assumed from README files. The endpoint, the expected response, and the Host header behavior were all confirmed against the real deployment.

For PostgreSQL instances, pg_isready is the canonical liveness probe. For Redis/Valkey, redis-cli ping or valkey-cli ping. For HTTP services, find the lightest public endpoint that doesn't require authentication. If the app validates Host headers, add the override — don't widen the allowlist.

The initialDelaySeconds and failureThreshold values vary per service. Home Assistant needs 30+ seconds. Immich ML needs time for model loading. Authelia is fast. Copy-pasting the same probe config across all deployments is how you get false-positive restart loops.

Health probes in Kubernetes are the same principle as Azure's health monitoring: Application Gateway health probes, AKS node health, Cosmos DB connection validation — all require understanding what "healthy" means for the specific service, not a generic HTTP 200 check.

Zero NetworkPolicies on the Database Namespace: The Gap That Let Any Pod Reach Authelia's Postgres

david — Fri, 17 Jul 2026 09:53:29 +0000

After locking down Vault's namespace with NetworkPolicies, I ran the same check on every other namespace. Everything runs on a BMAX Mini PC* behind a MikroTik RB5009* — one misconfigured NetworkPolicy and the wrong pod reaches Authelia's database. kubectl get networkpolicy -n database returned empty. Zero policies. Any pod anywhere in the cluster could reach Authelia's PostgreSQL and Redis instances with no network-layer restriction.

This is the same gap class as Vault — missed in that pass because the audit focused on external-facing trust roots first. The database namespace holds Authelia's actual session and configuration storage. If Authelia is the gate for every exposed service in the homelab, the database namespace is what Authelia trusts. A compromised workload in the apps namespace could brute-force session tokens directly at the Postgres level, bypassing Authelia's own API entirely.

View the complete homelab infrastructure source on GitHub 🐙

The Traffic Audit

Before writing any policy, I checked what actually talks to the database namespace. Guessing at traffic patterns is how you lock out a critical component and create a cascading failure.

Real traffic sources, confirmed against the live cluster:

Authelia (apps namespace): connects to postgres-authelia-rw:5432 (session/config storage) and redis-authelia:6379 (session cache). Confirmed via authelia/configmap.yml's storage and session host fields.
Prometheus (monitoring namespace): scrapes :9187 metrics from CNPG instances via the PodMonitor.
CNPG operator (cnpg-system namespace): calls each instance's status API on :8000 for health and failover decisions. Confirmed via the live pod's declared ports: postgresql:5432, metrics:9187, status:8000.

That's it. Three sources, three ports. The policy follows directly:

# kubernetes/system/postgres/network-policies.yml

# Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
---
# Allow same-namespace traffic (CNPG replication, operator-to-instance)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-intra-namespace
  namespace: database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: database
---
# Authelia → Postgres + Redis
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-apps
  namespace: database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: apps
      ports:
        - port: 5432
          protocol: TCP
        - port: 6379
          protocol: TCP
---
# Prometheus → CNPG metrics
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-monitoring
  namespace: database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 9187
          protocol: TCP
---
# CNPG operator → instance status API
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-cnpg-operator
  namespace: database
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: cnpg-system
      ports:
        - port: 8000
          protocol: TCP

Five policies. Each one corresponds to a real, verified traffic source. No guesswork.

What I Deliberately Left Alone

The cnpg-system namespace — where the operator itself runs — does not get a default-deny in this pass. This was a conscious decision, not an oversight.

The CNPG operator serves the admission webhook (cnpg-webhook-service:443) that kube-apiserver calls for every Cluster and Backup CRD operation cluster-wide. In k3s, apiserver-to-pod traffic doesn't originate from a normal namespaced pod IP. A default-deny in cnpg-system risks silently breaking webhook admission for all CNPG resources — every Cluster and Backup apply would fail.

On top of that, the operator already had a flaky restart history: 299 restarts over 21 days, with the most recent about 2 hours before this audit. Adding a NetworkPolicy that might interfere with its admission webhook, on a component that's already unstable, is how you turn a monitoring gap into an outage.

Flagged as a separate reliability issue. Not guessed at.

The Pattern

The audit followed the same methodology as the Vault fix:

kubectl get networkpolicy -n <namespace> — confirm it's empty
kubectl get pods -n <namespace> — identify what's running
Check each pod's declared ports and cross-reference with the consumer's config
Write policies that match real traffic, not assumed traffic
Apply, test, verify — don't apply and hope

The key discipline: never write a NetworkPolicy based on what you think the traffic pattern is. Check the actual config files, check the live pod's ports, and verify the source pod exists in the source namespace before committing.

Network segmentation at the Kubernetes layer — namespace-level NetworkPolicies, pod-level isolation, deny-by-default — is the same model as Azure NSGs at the subnet level, Hub-Spoke traffic flow, and Zero Trust network architecture. The scale changes, the principle doesn't.

DEV Community: david

My Terraform Runner Destroyed Itself Mid-Apply

The Setup

The Kill Shot

Why terraform plan Doesn't Catch This

The Fix: Move Atlantis Off k3s

The Ansible Layer

The Pattern

The ZFS ARC Freeze: How a Marginal PSU Killed My Entire Homelab

The Symptoms

The PSU Theory

The ZFS Deadlock

The Fixes

Fix 1: Disk I/O Isolation

Fix 2: ZFS ARC Limits

Fix 3: txg Timeout Tuning

Fix 4: Staggered Boot Order

What It Taught Me

How a Single Volume Was 65% of My Velero Backup and What I Almost Excluded Instead

The Symptom

The Investigation

The First Hypothesis (Wrong)

The Right Fix

The Exclusion

Result

The Pattern

How to Find Redundant Backups

🚀 Azure Firewall - Enterprise Forced Tunneling Edition — €49

Why a Cloud Backup Sync Was Failing Daily for Six Weeks

What the Audit Sweep Found

The Wrong First Conclusion

The Fix That Turned Out to Be Wrong

The Same Log Pattern, Two Opposite Conclusions

What This Argues For

MikroTik vs Ubiquiti for Home Network 2026

TL;DR — The Verdict

Quick Comparison

Category-by-Category Breakdown

Design & Build Quality

Features & Flexibility

Ease of Use

Security & Firewall

VPN

VLAN Support

Terraform & Automation Support

Ecosystem & Expandability

Price & Value

Who Should Buy Which

Frequently Asked Questions

Can MikroTik and UniFi devices coexist on the same network?

Is MikroTik really harder to set up than Ubiquiti?

Should I use MikroTik for my home Wi-Fi?

How does MikroTik security compare to Ubiquiti?

Can I manage MikroTik with Terraform?

What is the best MikroTik router for a home lab?

Conclusion

🚀 Azure Zero-Trust Starter Pack — €99

Best NAS for Home Backup 2026

TL;DR — The Picks

Selection Criteria — What Matters in a Home Backup NAS

Individual Reviews

1. Synology DS224+ — Best Overall

2. QNAP TS-264 — Best Features

3. Terramaster F2-223 — Best Budget

4. Synology DS423+ — Best for Growth

5. UGREEN DXP2800 — Best Newcomer

Head-to-Head Comparison

Buyer's Guide

Frequently Asked Questions

How much storage do I need for home backup?

What drives should I put in a NAS?

Is RAID the same as backup?

Should I use Synology Hybrid RAID or standard RAID?

Can I run Docker on these NAS devices?

How long does a NAS last?

Is a NAS worth it versus just using an external hard drive?

Conclusion

🚀 Azure Zero-Trust Starter Pack — €99

Best Mini PC for Homelab 2026

TL;DR — The Picks

Why `terraform plan` Doesn't Catch This