DEV Community: Kanu Kingsley

The 333 GB I didn't know I was keeping

Kanu Kingsley — Sun, 26 Jul 2026 19:04:50 +0000

My main storage array is 839 GB of btrfs sitting on a mdadm RAID5. Last month it was reading half full and I couldn't tell you what half. That bothered me more than a full disk would have, because a full disk at least tells you where the space went.

So I went looking. Twenty minutes later I had my answer: 333 GB of backups from 2024 that I had not touched since, plus a scatter of empty share definitions pointing at folders that no longer existed. I deleted the lot in about seven minutes. The interesting part is not the number. It's that the space had been invisible, and that OpenMediaVault makes the cleanup safe only if you do it the way OMV wants.

Inventory before you touch anything

The rule I hold to on storage is boring on purpose: look before you delete, and write the looking down. Two commands did most of it.

du -h --max-depth=3 /srv/dev-disk-by-uuid-*/ | sort -rh | head -40
find /srv -type d -empty

The first gives you the fat directories ranked by size, three levels deep, which is usually deep enough to name the culprit without drowning in output. The second finds the empty ones, which on an OMV box are often the tell-tale of a share whose data got moved or deleted while the share definition stayed behind.

I put the output in a text file and went through it line by line, tagging each entry as one of three things. Active, meaning something reads or writes it and it stays. Stale, meaning it is real data nobody needs anymore, like those 2024 backups. Orphan stub, meaning OMV still has a shared-folder entry for it but the underlying directory is gone. The 333 GB was almost all stale. The orphan stubs were tiny on disk but they are the part that bites you later, so they went on the list too.

Why I don't hand-edit OMV's config

Here is the thing that catches people. OpenMediaVault keeps its entire configuration in one XML file at /etc/openmediavault/config.xml, and it is right there, readable, editable. You can open it, find the share you want gone, delete the block, save. It will work until the next time OMV regenerates config from that file, which it does through its salt-based backend whenever you change almost anything in the web UI. Your hand edit either gets clobbered or, worse, leaves the running system and the config file disagreeing about what exists.

The supported path is omv-rpc, the same RPC interface the web UI drives. Anything the UI can do, omv-rpc can do from the shell, and it goes through the same validation and the same salt apply. So the destruction gets scripted, and OMV stays the source of truth about its own state.

Enumerate the shared folders first so you are working from UUIDs, not names:

omv-rpc -u admin 'ShareMgmt' 'enumerateSharedFolders' | jq -r '.[] | [.uuid, .name, .reldirpath] | @tsv'

Then, per target I had already confirmed as stale or orphan, delete the share definition by its UUID and apply:

omv-rpc -u admin 'ShareMgmt' 'delete' '{"uuid": "<the-uuid>"}'
omv-rpc -u admin 'Config' 'applyChanges' '{"modules": [], "force": false}'

The actual bytes I removed with a plain rm -rf on the confirmed-stale directories, one at a time, after the share definitions were gone. No script looped over a wildcard. Every deletion named one specific path I had already tagged by hand.

The sentinel UUID gotcha

OMV uses one special UUID, fa4b1c66-ef79-11e5-87a0-0002b3a176b4, as its "not set" reference. When a service, an NFS export, or an SMB share points at a shared folder, it stores that folder's UUID. Delete the folder definition out from under a still-referencing export and OMV does not always stop you, and the reference can fall back to that sentinel. You end up with an NFS export or an SMB share that resolves to nothing, and the symptom is a client mount that hangs or 404s rather than an error at delete time.

The order that avoids it: remove the referencing service first (the NFS export, the SMB share), then the shared-folder definition, then the data. I got this backwards on one entry, saw an export left pointing at the sentinel, and had to walk it back.

What it actually cost

Thirty minutes of inventory. Seven minutes of deletion. The imbalance is the whole lesson. The dangerous work was fast and the safe work was slow, which is the correct ratio and the opposite of how it feels in the moment when you can see 333 GB you want back.

I have put a reminder in the calendar to do this once a year. Not a cron job that deletes things, I am not automating rm -rf against my own data on a schedule. Just the inventory half, the du and the find, so once a year I have to look at what the array is actually holding and decide, on purpose, what stays.

Most homelab storage fills with ghosts. Old backups, half-finished migrations, shares for services you retired. The disk never complains until it is full, and by then the archaeology is harder. Cheaper to look every year while you still remember what the folders were for.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The bond that negotiated with nobody

Kanu Kingsley — Sun, 26 Jul 2026 19:03:30 +0000

I planned a 2-gigabit bond for my Proxmox host. For weeks it quietly ran at 1, and nothing in the logs told me why.

The design that looked right

The host has four network ports. My plan was textbook: pair them into two LACP bonds, then put an active-backup bond on top of both for redundancy. If a single link died, its bond would drop it. If a whole bond died, the top layer would fail over to the other. On a whiteboard it was clean.

The problem was the switch. Mine is a small smart-managed TP-Link, and "smart-managed" sounds like it speaks LACP. It doesn't. It only does static link aggregation, the kind where you hard-assign ports to a group and trust that both ends agree. LACP is the dynamic version where the two ends negotiate the bundle between themselves, and this switch has no idea how to answer that conversation.

The tell was a field full of zeros

Because the switch never answered, every LACP negotiation packet my host sent went nowhere, with no error and no log line to show for it. The failure surfaced in exactly one place: the bond's own status, where the partner MAC address read 00:00:00:00:00:00. All zeros is the network's way of saying nobody on the other end responded.

The effect was that only one of the two links in each bond ever came up. I had built for 2 gigabits of aggregate bandwidth and was quietly running at 1, and I would not have caught it without reading /proc/net/bonding/bond0 by hand.

The fix was to stop being clever

I tore the nested design down to a single active-backup bond over two ports. One active link with one standby, and immediate failback to the primary when it recovers. The config got shorter, and the failover I actually wanted still works. The 2-gigabit number was never reachable with this switch, so collapsing the design lost me nothing real.

The gotcha on the way out

The change itself took one more fight. Linux will not switch a bond's mode while it still has slave interfaces attached. Try it and the kernel refuses the write with "Directory not empty," and ifreload swallows that refusal rather than surfacing it, so the config reads as applied when it isn't. The bond keeps its old mode and you lose time wondering why. The fix is to tear the bond all the way down and bring it back up with ifdown bond0 && ifup bond0, so the device is recreated fresh and the mode is set before any slave attaches. That cost me half an hour.

What I took from it

Match the topology to what your switch can actually do. LACP everywhere is a cargo-cult default that assumes 802.3ad-capable gear on both ends of every link. If yours only does static aggregation, a plain active-backup bond gives you reliable failover without chasing a bandwidth number you were never going to reach. And when a change looks applied but the behavior refuses to move, check whether the tool quietly ate an error for you.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The recovery path I never had

Kanu Kingsley — Sun, 26 Jul 2026 19:02:31 +0000

Last week I wrote about bringing an HP DL380 G7 back from fifteen months of cold storage. What I did not say was how close I came to bricking my own recovery.

The homelab had a bootstrap network I had set up in 2024 called 10.99.110.x. Nothing lived on it except the host's secondary IP and the NAS guest's second interface. When I set up VLAN 100 tagged as the new SERVERS network, I retired 10.99.110.x. Between the cutover and the moment pfSense started routing VLAN 100 correctly, my only management path to the box was that dying network. If pfSense had glitched mid-cutover, I would have been at the chassis with a keyboard.

That gap is what showed me what VLAN 200 MGMT is actually for. It also eventually pulled VLAN 300 DMZ out of me. Three VLANs today, each with a specific job.

SERVERS is where the work happens

VLAN 100 is 10.99.99.0/24. Every LXC, every VM, the PVE host's management IP. Pi-hole, Traefik VRRP VIP at .29, Nextcloud, GitLab, Authelia, Vaultwarden, the observability stack. It is the network that all my other posts talk about.

If SERVERS is broken, I need a way in that does not depend on SERVERS.

MGMT is where the recovery lives

VLAN 200 is 192.0.2.0/24. Very few things live on it. The host has a secondary IP at 192.0.2.5. iLO 3 lives at 192.0.2.3. My laptop drops on one of the MGMT-tagged ports on the switch and gets a MGMT address via DHCP.

Nothing on this VLAN has a default gateway. The whole point is that it does not route out. If SERVERS goes down, or if pfSense itself is broken, I can still direct-L2 to 192.0.2.5, log in over SSH, and see what happened.

iLO 3 also deserves its own segment for a different reason. Its firmware last shipped in December 2020. It only speaks TLS 1.0, only accepts DSA-1024 SSH keys, and answers IPMI/DCMI on UDP/623. That stack has no business being reachable from the wider LAN. On MGMT, only my laptop and the host itself can talk to it, and the pfSense rules between SERVERS and MGMT make sure nothing on SERVERS can escalate through it.

DMZ is where the internet gets in

VLAN 300 is 203.0.113.0/24. One IP is in use. 203.0.113.10 is my cloudflared LXC.

Cloudflared holds the outbound TLS session to Cloudflare's edge. Requests to cloud.iamkay.eu, ha.iamkay.eu, and a handful of other public routes flow inbound through that tunnel, hit cloudflared, and cloudflared forwards them to the Traefik VRRP VIP at 10.99.99.29 on port 443. The pfSense rule that allows this flow is one line:

src=203.0.113.10 -> dst=10.99.99.29 tcp/443

Nothing else can cross from DMZ to SERVERS. Cloudflared cannot SSH into anything, cannot hit Vaultwarden directly, cannot touch Nextcloud's data volume. It has exactly one job.

Enterprise DMZ patterns describe a /24 subnet full of load balancers, WAFs, and reverse proxies, all serving multiple internet-facing apps. That is the right model at that scale. At homelab scale, my entire DMZ is one process and one address. The rule set can be minimal because the surface is minimal.

The rules that tie it together

pfSense is the only device that can route between VLANs. The rules I care about most:

SERVERS to MGMT is blocked with no exceptions. I verified this six different ways before trusting it.
DMZ to SERVERS allows exactly 203.0.113.10 -> 10.99.99.29 on tcp/443. Nothing else.
MGMT to anywhere is allowed, because MGMT is my hands. If someone is on MGMT they either have physical access to the switch or my laptop.
SERVERS to the internet is allowed for outbound traffic: updates, DNS forwarding, mail relay, ACME challenges.

The switch itself is a TP-Link TL-SG108E v1. Static VLAN tagging, not 802.3ad. That kept me from doing LACP with pfSense, which is a whole other story I will get to. The host runs a single active-backup bond over two 1GbE NICs. VLAN tags terminate on vmbr0.100, vmbr0.200, and vmbr0.300 sub-interfaces.

Tradeoffs I know I have

Family devices are on the untagged VLAN 1 today. That is not a decision, it is an omission. When I move them to their own VLAN, the pfSense rules need to say family devices can reach cloud.iamkay.eu and nothing else on the homelab. That work is queued and not urgent. Right now the family is one household and I trust the trust boundary.

I do not have a full L3 switch. The TL-SG108E does static VLAN tagging fine, but everything routes through pfSense. If pfSense is a single point, MGMT is the fallback. If both are down at once, I am at the chassis.

What it comes back to

Each of the three VLANs is named for the thing it does. SERVERS holds the workloads. MGMT is how I reach them when SERVERS is down. DMZ is one process talking to the internet. The structure was not something I designed up front. It came out of asking, at every layer, what happens if this network dies.

The cold-storage recovery would have gone sideways if the box between my laptop and pfSense had died during the VLAN cutover. It did not, but that near-miss is what taught me the shape of MGMT. MGMT refuses to route anywhere. DMZ routes exactly one thing. Both segments exist to answer what happens when the primary path breaks.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

I never ran ESXi in production

Kanu Kingsley — Sun, 26 Jul 2026 15:46:06 +0000

Most "why Proxmox" content in 2025-2026 is a migration story driven by Broadcom's ESXi pricing changes. The author had a working VMware stack and got priced out. I'm not that author. I evaluated both, picked Proxmox in 2024, and built on it without ever running ESXi in production. Two years in, I'd make the same call.

It reads as either incompetent or contrarian until the rest of the post lands. Here's the reasoning.

The three reasons it was the easy call

1. LXC and KVM in one host

Most workloads in this homelab are LXCs. Pi-hole, Vaultwarden, Authelia, Traefik, the monitoring stack, GitLab CE itself, all containers sharing the host kernel. A few things need full VM isolation (the NAS guest, Proxmox Backup Server, the Home Assistant OS appliance). Same hypervisor, same CLI, same web UI for both shapes of workload.

The alternative is ESXi for the VMs and a separate toolchain (containerd, Docker, Kubernetes, take your pick) for the containers. That's two backup pipelines, two HA stories, two places for config drift to surprise you at 2 AM.

pct exec 254 systemctl status authelia and qm start 189 are the same shape. New hires don't have to learn one tool for containers and a different one for VMs.

2. Proxmox Backup Server beats the free Veeam alternative

Chunk-level deduplication. Backups across guests and across time share storage. A nightly backup of all 11 LXCs and 2 VMs runs in about ten minutes and adds a few hundred MB of new chunks, because most of the content is the same as yesterday.

Cluster-scheduled. One job definition runs across every node in the cluster. No per-node cron, no manual rotation when a node moves.

Restore to a different storage class. A backup taken from local-lvm on the G7 restores onto ZFS on a G5 cluster node without conversion gymnastics.

Veeam Community Edition is the free comparison. It works. It also caps repository size, doesn't dedup at the chunk level, and lacks the cluster-aware scheduling that makes PBS feel like a built-in feature rather than a bolt-on.

When backups are the load-bearing wall of a homelab, the part that lets me say yes to family data later, the free option that compromises least is the one that wins.

3. Cluster and HA without vCenter

ESXi clustering requires vCenter Server. vCenter is its own VM, its own database, its own patching cadence, its own license. The minimum production footprint to get HA is three ESXi hosts plus one vCenter plus shared storage. The vCenter VM itself also needs to be backed up off the cluster it manages.

Proxmox does the same job from the per-node web UI. pvecm create, pvecm add from each new node, edit corosync.conf if the auto-detect picks the wrong network, done. HA failover policy is ha-manager add ct:256 --group g5-only. No separate management plane.

Three-node cluster running. A real workload migrated to a different node on the night I shut down one of the cluster members. Total observability downtime: about a minute and a half.

Honest tradeoffs

I want this post to read as opinionated, not as marketing.

Ecosystem. VMware has decades of third-party integrations across storage vendors, backup vendors, networking overlays, and security tooling. Proxmox has fewer. If you need a specific compliance-certified storage appliance or a vendor-specific tooling chain, VMware is sometimes the only choice.

Audit logging. Proxmox logs the basics. Deep audit trails with role hierarchy, per-action attestation, and tamper-evident logs are an ESXi + Aria stack feature that doesn't have a Proxmox equivalent. For SOC 2, HIPAA, or FedRAMP work this matters. For a homelab and a small team it doesn't.

Support. The Proxmox Enterprise subscription exists and is reasonable, but the support contract isn't the same as VMware's. For 24/7 white-glove production support, ESXi still wins. For the small-team case, the community forums and the wiki are usually faster anyway.

The honest summary: at small scale Proxmox wins on tooling unification, on free-tier capability, and on the speed at which you can change your mind. At enterprise scale the tradeoffs reverse.

What kept me here

I haven't reconsidered in two years because the daily friction is low. The web UI does what I want, the CLI is consistent across LXC and VM, and the backup story stays one tool. Migrating a service between nodes is two commands; when a node falls over the cluster handles it.

The homelab keeps growing. Every additional service is a new LXC or VM in the same pane of glass. The platform hasn't forced a new toolchain on me to keep up.

The right tool depends on the scale you actually operate at. For mine, it's Proxmox.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.