DEV Community: Kingsley Kanu

The runner that was busy on someone else's job

Kingsley Kanu — Thu, 13 Aug 2026 08:00:00 +0000

My CI pipeline sat pending for ten minutes. The runner was healthy. So was every other pipeline in the queue. The default setting was the bug.

Healthy and stuck at the same time

GitLab Runner ships with concurrent = 1 in /etc/gitlab-runner/config.toml. That one line quietly makes every pipeline run single-file across every project sharing that runner. Not per project. Across all of them. So a chatty repo pushing every few minutes can starve everything else, and the symptom is the least helpful one possible: your job sits pending while gitlab-runner status cheerfully reports "healthy."

Healthy and stuck at the same time is the tell. The runner is not broken. It is busy, on something that is not yours.

The diagnosis

The check that confirmed it runs inside the runner's container:

docker ps --filter name=runner-
docker inspect <build-container> --format '{{.Config.Env}}'

Read CI_PROJECT_PATH and CI_PIPELINE_ID off the running build container and you learn exactly which project is holding the single lane. In my case it was a different repo entirely, mid-build, with mine queued politely behind it.

The fix and the trade

The fix is one number: bump to concurrent = 2, or higher, watching RAM, because every extra lane is another set of job containers running at once. Back up the config first, restart the runner, and the pending pipelines unblock within about thirty seconds.

The broader lesson is about defaults. concurrent = 1 is tuned for one project per runner, which is the simplest possible case and rarely the real one. The moment your runner serves several repos, that default stops protecting you and starts throttling you. "Healthy but not moving" is the signature of a default you have outgrown, not a component that has failed.

The hole I never had to open

Kingsley Kanu — Tue, 11 Aug 2026 08:00:00 +0000

My family cloud is reachable from anywhere on the internet, and I never opened a single firewall port to do it.

The hole I did not want

For years the default answer to "how do I reach my home server from outside" was port forwarding: poke a hole in your router, expose a service to the whole internet, and hope your patching keeps up with the people scanning for it. I did not want that hole. So there is not one.

How the tunnel replaces it

Instead I run cloudflared, Cloudflare's tunnel agent, in a small container on an isolated DMZ segment. It makes an outbound TLS connection to Cloudflare's edge and holds it open. When someone requests cloud.iamkay.eu, the request arrives at Cloudflare, travels back down that already-open tunnel to cloudflared, and cloudflared hands it to my internal reverse proxy, which routes it to Nextcloud. The connection is always outbound from my side. My router's inbound rules never change, because the answer to "who is allowed in" stays nobody.

The full path is worth spelling out: phone on mobile data, to Cloudflare's edge, down the tunnel to cloudflared, to the Traefik virtual IP, to Nextcloud, to the NAS behind it. No port forward anywhere in that chain.

The blast radius is one line

The part I like most is how small the exposure is. cloudflared lives on its own DMZ VLAN, and the firewall rule letting it reach the internal network is exactly one line: source 203.0.113.10, destination the proxy virtual IP at 10.99.99.29, port 443, nothing else. If cloudflared were ever compromised, it cannot SSH anywhere, cannot reach the vault, cannot touch Nextcloud's storage directly. It has one job and the firewall enforces exactly that job.

It is free for personal use, and you get Cloudflare's DDoS protection and WAF in front of your services as a bonus. If you are still port forwarding self-hosted services in 2026, this is the switch to make.

A note on the addresses

The IP addresses and hostnames in this post are documentation placeholders on reserved ranges, not my real ones. The public route cloud.iamkay.eu is real because it is a destination I want people to reach; the internal IPs are stand-ins on purpose. The architecture is exactly how I run it. Nothing here points at a live target.

The failover that fooled me

Kingsley Kanu — Thu, 06 Aug 2026 17:23:37 +0000

My reverse proxy had VRRP failover. I killed the process directly to test it, and the traffic black-holed anyway.

The setup that looked complete

I run a Traefik pair with keepalived holding a shared virtual IP between them. The idea is simple and standard: if the active node dies, the IP moves to the standby and traffic keeps flowing. And it does, if the whole node dies. Pull the power on the master and the VIP is on the backup in a couple of seconds.

The gap is what happens when the box lives but the application does not.

Node up, app down, VIP stuck

keepalived on its own watches only the VRRP heartbeat between the two peers. That heartbeat proves the box is alive and on the network. It says nothing about whether the application on that box is actually serving. So when I killed the Traefik process but left the machine running, the heartbeat kept ticking, keepalived saw a healthy master, and the virtual IP sat happily on a node that was no longer answering a single connection. The failover I thought I had covered exactly the failure that is least likely (a whole machine dying) and missed the one that is most likely (a process crashing).

The five lines that fix it

The fix is a health check that keepalived runs on a timer and folds into its own priority:

vrrp_script chk_traefik {
  script "curl -sf http://localhost/ping || exit 1"
  interval 2
  fall 2
  rise 3
}

Reference it from the vrrp_instance track_script block, and now keepalived lowers its own priority the moment the app stops answering, which hands the VIP to the standby. The same pattern works for nginx, haproxy, or any single process sitting behind a virtual IP.

What it comes back to

The heartbeat proves the box is up. The track_script proves the app is up. If your VIP fronts an application, you need both, and the only reliable way to know you have both is to test by killing the process, not the machine. The machine-death test passes on a broken config. The process-death test is the one that tells the truth.

My passwords no longer leave my house

Kingsley Kanu — Thu, 06 Aug 2026 17:22:46 +0000

My password vault runs on a box in my house now. Every credential I have syncs to my phone and my browser the same way it did when I paid Bitwarden for it, except the encrypted database lives on hardware I can walk over and touch.

That sentence is either reassuring or terrifying depending on how you feel about being your own backup department, and I want to be honest about both.

What it actually is

Vaultwarden is a reimplementation of the Bitwarden server in Rust. It speaks the same API as the official server, which is the whole point: the official Bitwarden browser extension, the mobile app, and the desktop client all work against it unchanged. You point them at your own server URL instead of bitwarden.com and they cannot tell the difference. The clients stay official and audited. Only the server is swapped.

The reason to run Vaultwarden instead of the official self-hosted Bitwarden server is footprint. The official server is a stack of containers and wants a real chunk of RAM. Vaultwarden is a single process with a SQLite database, sitting at around 50 MB resident. It runs in a Docker container inside an LXC on my cluster and I mostly forget it exists. It lives behind Traefik, which terminates TLS, so the clients reach it at https://vault.hm.example.com with a proper certificate.

The part that is not optional

Here is where self-hosting a vault stops being a fun weekend project and becomes a responsibility. That SQLite file, db.sqlite3, is the vault. Not a cache of it, not a copy, the actual encrypted store of every password I have. If I lose it and have no backup, I have locked myself out of my own life. Bitwarden's hosted service handles this with geo-redundant storage across regions. When you self-host, that is now your job.

So the backup discipline came before I trusted a single real password to it. Proxmox Backup Server takes a daily snapshot of the whole container. That covers hardware failure, a bad upgrade, a fat-fingered delete. It does not cover the house burning down, and a backup that only exists in the same building as the original is one accident away from not existing. The next step, and the one that actually brings this to parity with a hosted service, is an encrypted off-site copy. I am pushing the backup to Backblaze B2 through rclone with client-side encryption, so the copy that leaves the house is already ciphertext before it hits the wire and I hold the only key.

Why it is not behind my SSO

I run Authelia in front of most of my internal services, and Vaultwarden is deliberately not one of them. The Bitwarden browser extension and mobile app talk to the server's API directly on every sync. A forward-auth proxy answers unauthenticated requests with an HTML login page, which the extension has no idea what to do with, so it just fails to sync.

Vaultwarden does not need Authelia in front of it anyway. The vault is end-to-end encrypted with a master password the server never sees, admin registration is disabled, and Bitwarden's own brute-force throttle is on. Wrapping it in a second login would break the clients to protect data that is already encrypted at rest with a key I never upload.

The tradeoff, stated plainly

I traded roughly $40 a year and Bitwarden's geo-redundant infrastructure for full control of where my most sensitive data lives, at the cost of owning the backup risk myself. That trade is only worth it if the backup discipline is real. A self-hosted vault with no off-site copy is not cheaper than Bitwarden, it is just riskier and free, which is a bad deal dressed up as a good one.

Paired with daily PBS snapshots and an encrypted off-site copy, I am comfortable calling it parity. The vault is mine, the clients are the same ones I already trusted, and the only thing I added was the obligation to take backups seriously.

A note on the addresses

The internal hostname in this post, vault.hm.example.com, is a documentation placeholder on a reserved example domain, not the real name on my network. Any IP addresses elsewhere in this series use the ranges set aside for documentation. The architecture is real. The specific names and addresses are stand-ins on purpose, so nothing here maps to a live target.

The DL380 had been off for fifteen months

Kingsley Kanu — Mon, 03 Aug 2026 09:53:44 +0000

I powered down an HP DL380 G7 in early 2025 and did not touch it for fifteen months. It sat in a closet through a job change. On 27 May 2026 I went back to it.

Would the PSUs hold. Would the disks spin clean. Would Proxmox boot. Would I remember the passwords.

First boot

Front panel button. Fans spun up. POST stopped on a memory training failure: one DIMM in CPU socket 1, "uncorrectable error during training."

Almost opened the chassis to reseat. Waited ten minutes first. A forum post had warned me cold-stored hardware needs a few minutes to settle. Power cycled. Clean on the second attempt. No memory complaints in any boot since.

Wait through the first symptom once before you touch anything.

Proxmox came up easily

F11 at POST. Picked the boot SSD on the P410i. Proxmox came up on kernel 6.5.13-3-pve. Root password I still remembered, typed at the USB keyboard plugged into the back of the chassis.

First job at that console: move the host onto the current network. The VLAN and address from 2024 were gone. Edited /etc/network/interfaces over vi, ran ifreload -a, watched the new bridge come up, walked back to the laptop. SSH worked first try.

From there everything ran over SSH. qm list returned twenty-two VMs and one LXC. None running. Past-me had set onboot=0 on every guest before the shutdown, which is a discipline I have not always had.

Started the NAS (qm start 189), let OpenMediaVault bring up its NFS exports, checked cat /proc/mdstat:

md0 : active raid5 sdd[2] sdc[1] sdb[0]
      838860288 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid5 sdh[3] sdg[2] sdf[1] sde[0]
      1431655936 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

Both arrays clean. The btrfs filesystems on top were readable. Fifteen months of cold did nothing to the storage layer.

iLO was the hard half.

The iLO problem

I wanted remote console access. The DL380 G7 has iLO 3, HP's out-of-band management processor from 2010. The final firmware shipped in December 2020, version 1.94 (HPE iLO firmware list).

iLO 3 v1.94 only negotiates TLS 1.0. Chrome returned ERR_SSL_VERSION_OR_CIPHER_MISMATCH with no override. Edge same answer. Firefox connected after I set security.tls.version.min = 1 in about:config. Fresh Firefox profile, dedicated to this BMC and nothing else.

The TLS workaround I had. The bigger problem was the iLO IP I had configured in 2024 was on a VLAN that no longer existed in 2026. Admin password I still remembered. I just had no route to the login page. The Proxmox host had a current address, but iLO sat on a dedicated NIC behind one that nothing on the current network could speak to.

In-band IPMI saved me a chassis open.

In-band IPMI

iLO speaks IPMI 2.0 over a KCS (Keyboard Controller Style) channel that the host OS sees as /dev/ipmi0. Root on the host gives you admin access to the BMC. No BMC password needed; you already proved you have the keys to Linux.

Debian 12 needed ipmitool. The host had no internet (network was still half-rebuilt), so I sideloaded the .deb stack from the laptop: ipmitool 1.8.19-4+deb12u2 plus freeipmi-common 1.6.10-1 and libfreeipmi17 1.6.10-1+b1, all SHA256-verified. Then:

$ ipmitool user list 1
ID  Name             Callin  Link Auth   IPMI Msg   Channel Priv Limit
1                    true    false       false      Unknown (0x00)
2   kay              true    true        true       ADMINISTRATOR

Admin in slot 2 (slot 1 is the IPMI-spec empty placeholder). Always list before you write.

Rotated the admin password. The recorded one still worked. Cheap insurance.

$ ipmitool user set password 2 'a-new-strong-password'
$ ipmitool channel setaccess 1 2 callin=on ipmi=on link=on privilege=4

The actual rescue was the IP. LAN on iLO 3 is channel 2, not 1. Channel 1 errors out, and anything you read online about "channel 1 LAN" is iLO 4 or non-HP.

$ ipmitool lan set 2 ipsrc static
$ ipmitool lan set 2 ipaddr 10.99.99.3
$ ipmitool lan set 2 netmask 255.255.255.0
$ ipmitool lan set 2 defgw ipaddr 10.99.99.1
$ ipmitool mc reset cold
Sent cold reset command to MC

The cold reset is where I burned forty minutes the first time. iLO 3 has gratuitous ARP disabled. Changing the IP does not make the BMC announce its new binding, and the upstream switch keeps the stale ARP entry for the old IP. ipmitool mc reset cold power-cycles the BMC's network stack, which is what forces the announce. Without it you watch pings drop, certain one of your lan set commands was rejected, when in fact all four were accepted just fine.

One more iLO 3 gotcha. The BMC does not log IPMI admin operations to the SEL that ipmitool sel list returns. The audit trail lives in HP's Integrated Management Log, reachable only through the web UI, RIBCL, or hponcfg, and RIBCL needs the same TLS handshake modern OpenSSL refuses. So you do recovery work and there is no on-host record. Keep your own log.

What it taught me

iLO was harder than Proxmox. That ratio will hold for any pre-2014 enterprise server. Hypervisor and storage are still maintained. The BMC is a frozen artefact of whatever firmware shipped before the vendor stopped caring.

Treat it like that. BMC on its own VLAN, no internet path. Credential on paper outside the BMC; recovery is in-band IPMI, which needs the host up. Cold-start drill once a year. Starting from scratch today I would buy a newer server.

The box came back. Data intact. Three sessions over four days. Fifteen months of cold did nothing irreversible.

Resources

HPE iLO firmware history (Wikipedia)
ipmitool upstream
Where this fits: the DL380 G7 stays as temporary quorum until a 3-node EliteDesk G5 plus WRX90 Threadripper Pro cluster takes over. More on that build in a future post.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The eight login forms I deleted

Kingsley Kanu — Mon, 03 Aug 2026 09:52:16 +0000

For a while every self-hosted service in my homelab had its own login form. Eight services, eight password prompts, eight places a weak password could hide. That is not security, it is friction pretending to be security.

Authelia fixed it in about two hours. This is what it does, why I chose it over the heavier option, and the one class of service I deliberately left outside it.

How it works

Authelia sits behind Traefik as a ForwardAuth middleware. Every request to an internal service gets checked against Authelia first. One login sets a cookie scoped to my internal domain, and every service under that domain trusts the cookie. Log in once, you are through to all of them. TOTP is enforced for anything in the admin group, so the sensitive services always ask for a second factor.

The internal domain here is hm.example.com, a documentation placeholder. The real one is a private split-horizon domain that only resolves inside the house. The shape is what matters: one parent domain, a wildcard cookie, and every service living as a subdomain under it.

The default that matters

The design decision worth copying is the default. Authelia's access control starts at deny, and every service I want reachable is an explicit allow rule. Adding a service means adding a line. Forgetting to add a line means the service is locked, not open. That is the failure mode I want. A system that fails closed is one you can extend without lying awake wondering what you left exposed.

Why not Keycloak

People reach for Keycloak here, and for an organisation with real identity needs it is the right tool: federation, user self-service, a full admin console. For five users it is a lot of moving parts to run and patch. Authelia is a single Go binary with a file-based user database, argon2id password hashing, and a SQLite session store. Sessions expire after five minutes idle, one hour hard cap, with a one-month "remember me" for trusted devices. It does less, and for this scale that is the point.

The service I left outside

Forward-auth returns HTML login pages. That is fine for a browser, and wrong for a sync client. The Nextcloud mobile app, CalDAV, CardDAV, and git over SSH all speak their own auth on every call and have no idea what to do with a redirect to an HTML login form. Put them behind forward-auth and they fail silently.

So those services stay outside Authelia and lean on their own authentication: Nextcloud's brute-force throttle and app passwords, git's SSH keys. The rule I follow is to match the auth pattern to what the client can actually handle, rather than wrapping everything in the same layer because it feels more secure.

What it comes back to

Single sign-on for a small system is a one-day project, not a quarter-long identity programme. One gatekeeper, a deny-by-default policy, TOTP on the admin group, and a short list of services that stay outside because their clients cannot speak the protocol. That is enough to stop running eight separate password forms, which was the whole point.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The daemon that failed by doing nothing

Kingsley Kanu — Mon, 03 Aug 2026 09:51:30 +0000

My Proxmox backups had been failing silently for ten days. Not failing, actually. Not running at all, and nothing alerted on it.

The daemon that quietly stopped

A cluster shakedown turned up pvescheduler.service sitting inactive (dead) on one node since the last upgrade window. That daemon runs every scheduled backup, replication, and prune job on the host. When it is stopped, the daily job does not error. It does not fire. There is no failed task in the log, because no task was ever started. The system was, from its own point of view, perfectly healthy. It just was not doing the one thing I was counting on.

The gap that let it hide

Most monitoring checks systemctl is-active. That catches the state where a service is supposed to be running and has crashed: it was up, now it is down, page someone. What it does not catch is the state where a service is supposed to be running and was cleanly stopped and never came back. To is-active those look different, and the second one is where silent failures live.

The Prometheus rule that would have paged me on day one is short:

node_systemd_unit_state{state="active"} == 0
  and node_systemd_unit_state{state="enabled"} == 1

Enabled but not active. The unit is configured to run at boot, and right now it is not running. That combination is almost always a mistake, and it is the cheapest dead-daemon check there is. It would have caught this on the first morning instead of the tenth.

The fix, and the lesson

The fix once I found it took ninety seconds: systemctl start pvescheduler.service, and the next backup window ran all eleven guests. The fix is never the interesting part of one of these. The interesting part is the ten days.

is-active is table stakes for monitoring; everyone has it. enabled-but-inactive almost never is, and that gap is exactly where the quiet failures sit. If you run scheduled backups, go check right now that they actually ran recently, and then add the alert that would have told you without you having to look.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The 333 GB I didn't know I was keeping

Kingsley Kanu — Sun, 26 Jul 2026 19:04:50 +0000

My main storage array is 839 GB of btrfs sitting on a mdadm RAID5. Last month it was reading half full and I couldn't tell you what half. That bothered me more than a full disk would have, because a full disk at least tells you where the space went.

So I went looking. Twenty minutes later I had my answer: 333 GB of backups from 2024 that I had not touched since, plus a scatter of empty share definitions pointing at folders that no longer existed. I deleted the lot in about seven minutes. The interesting part is not the number. It's that the space had been invisible, and that OpenMediaVault makes the cleanup safe only if you do it the way OMV wants.

Inventory before you touch anything

The rule I hold to on storage is boring on purpose: look before you delete, and write the looking down. Two commands did most of it.

du -h --max-depth=3 /srv/dev-disk-by-uuid-*/ | sort -rh | head -40
find /srv -type d -empty

The first gives you the fat directories ranked by size, three levels deep, which is usually deep enough to name the culprit without drowning in output. The second finds the empty ones, which on an OMV box are often the tell-tale of a share whose data got moved or deleted while the share definition stayed behind.

I put the output in a text file and went through it line by line, tagging each entry as one of three things. Active, meaning something reads or writes it and it stays. Stale, meaning it is real data nobody needs anymore, like those 2024 backups. Orphan stub, meaning OMV still has a shared-folder entry for it but the underlying directory is gone. The 333 GB was almost all stale. The orphan stubs were tiny on disk but they are the part that bites you later, so they went on the list too.

Why I don't hand-edit OMV's config

Here is the thing that catches people. OpenMediaVault keeps its entire configuration in one XML file at /etc/openmediavault/config.xml, and it is right there, readable, editable. You can open it, find the share you want gone, delete the block, save. It will work until the next time OMV regenerates config from that file, which it does through its salt-based backend whenever you change almost anything in the web UI. Your hand edit either gets clobbered or, worse, leaves the running system and the config file disagreeing about what exists.

The supported path is omv-rpc, the same RPC interface the web UI drives. Anything the UI can do, omv-rpc can do from the shell, and it goes through the same validation and the same salt apply. So the destruction gets scripted, and OMV stays the source of truth about its own state.

Enumerate the shared folders first so you are working from UUIDs, not names:

omv-rpc -u admin 'ShareMgmt' 'enumerateSharedFolders' | jq -r '.[] | [.uuid, .name, .reldirpath] | @tsv'

Then, per target I had already confirmed as stale or orphan, delete the share definition by its UUID and apply:

omv-rpc -u admin 'ShareMgmt' 'delete' '{"uuid": "<the-uuid>"}'
omv-rpc -u admin 'Config' 'applyChanges' '{"modules": [], "force": false}'

The actual bytes I removed with a plain rm -rf on the confirmed-stale directories, one at a time, after the share definitions were gone. No script looped over a wildcard. Every deletion named one specific path I had already tagged by hand.

The sentinel UUID gotcha

OMV uses one special UUID, fa4b1c66-ef79-11e5-87a0-0002b3a176b4, as its "not set" reference. When a service, an NFS export, or an SMB share points at a shared folder, it stores that folder's UUID. Delete the folder definition out from under a still-referencing export and OMV does not always stop you, and the reference can fall back to that sentinel. You end up with an NFS export or an SMB share that resolves to nothing, and the symptom is a client mount that hangs or 404s rather than an error at delete time.

The order that avoids it: remove the referencing service first (the NFS export, the SMB share), then the shared-folder definition, then the data. I got this backwards on one entry, saw an export left pointing at the sentinel, and had to walk it back.

What it actually cost

Thirty minutes of inventory. Seven minutes of deletion. The imbalance is the whole lesson. The dangerous work was fast and the safe work was slow, which is the correct ratio and the opposite of how it feels in the moment when you can see 333 GB you want back.

I have put a reminder in the calendar to do this once a year. Not a cron job that deletes things, I am not automating rm -rf against my own data on a schedule. Just the inventory half, the du and the find, so once a year I have to look at what the array is actually holding and decide, on purpose, what stays.

Most homelab storage fills with ghosts. Old backups, half-finished migrations, shares for services you retired. The disk never complains until it is full, and by then the archaeology is harder. Cheaper to look every year while you still remember what the folders were for.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The bond that negotiated with nobody

Kingsley Kanu — Sun, 26 Jul 2026 19:03:30 +0000

I planned a 2-gigabit bond for my Proxmox host. For weeks it quietly ran at 1, and nothing in the logs told me why.

The design that looked right

The host has four network ports. My plan was textbook: pair them into two LACP bonds, then put an active-backup bond on top of both for redundancy. If a single link died, its bond would drop it. If a whole bond died, the top layer would fail over to the other. On a whiteboard it was clean.

The problem was the switch. Mine is a small smart-managed TP-Link, and "smart-managed" sounds like it speaks LACP. It doesn't. It only does static link aggregation, the kind where you hard-assign ports to a group and trust that both ends agree. LACP is the dynamic version where the two ends negotiate the bundle between themselves, and this switch has no idea how to answer that conversation.

The tell was a field full of zeros

Because the switch never answered, every LACP negotiation packet my host sent went nowhere, with no error and no log line to show for it. The failure surfaced in exactly one place: the bond's own status, where the partner MAC address read 00:00:00:00:00:00. All zeros is the network's way of saying nobody on the other end responded.

The effect was that only one of the two links in each bond ever came up. I had built for 2 gigabits of aggregate bandwidth and was quietly running at 1, and I would not have caught it without reading /proc/net/bonding/bond0 by hand.

The fix was to stop being clever

I tore the nested design down to a single active-backup bond over two ports. One active link with one standby, and immediate failback to the primary when it recovers. The config got shorter, and the failover I actually wanted still works. The 2-gigabit number was never reachable with this switch, so collapsing the design lost me nothing real.

The gotcha on the way out

The change itself took one more fight. Linux will not switch a bond's mode while it still has slave interfaces attached. Try it and the kernel refuses the write with "Directory not empty," and ifreload swallows that refusal rather than surfacing it, so the config reads as applied when it isn't. The bond keeps its old mode and you lose time wondering why. The fix is to tear the bond all the way down and bring it back up with ifdown bond0 && ifup bond0, so the device is recreated fresh and the mode is set before any slave attaches. That cost me half an hour.

What I took from it

Match the topology to what your switch can actually do. LACP everywhere is a cargo-cult default that assumes 802.3ad-capable gear on both ends of every link. If yours only does static aggregation, a plain active-backup bond gives you reliable failover without chasing a bandwidth number you were never going to reach. And when a change looks applied but the behavior refuses to move, check whether the tool quietly ate an error for you.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The recovery path I never had

Kingsley Kanu — Sun, 26 Jul 2026 19:02:31 +0000

Last week I wrote about bringing an HP DL380 G7 back from fifteen months of cold storage. What I did not say was how close I came to bricking my own recovery.

The homelab had a bootstrap network I had set up in 2024 called 10.99.110.x. Nothing lived on it except the host's secondary IP and the NAS guest's second interface. When I set up VLAN 100 tagged as the new SERVERS network, I retired 10.99.110.x. Between the cutover and the moment pfSense started routing VLAN 100 correctly, my only management path to the box was that dying network. If pfSense had glitched mid-cutover, I would have been at the chassis with a keyboard.

That gap is what showed me what VLAN 200 MGMT is actually for. It also eventually pulled VLAN 300 DMZ out of me. Three VLANs today, each with a specific job.

SERVERS is where the work happens

VLAN 100 is 10.99.99.0/24. Every LXC, every VM, the PVE host's management IP. Pi-hole, Traefik VRRP VIP at .29, Nextcloud, GitLab, Authelia, Vaultwarden, the observability stack. It is the network that all my other posts talk about.

If SERVERS is broken, I need a way in that does not depend on SERVERS.

MGMT is where the recovery lives

VLAN 200 is 192.0.2.0/24. Very few things live on it. The host has a secondary IP at 192.0.2.5. iLO 3 lives at 192.0.2.3. My laptop drops on one of the MGMT-tagged ports on the switch and gets a MGMT address via DHCP.

Nothing on this VLAN has a default gateway. The whole point is that it does not route out. If SERVERS goes down, or if pfSense itself is broken, I can still direct-L2 to 192.0.2.5, log in over SSH, and see what happened.

iLO 3 also deserves its own segment for a different reason. Its firmware last shipped in December 2020. It only speaks TLS 1.0, only accepts DSA-1024 SSH keys, and answers IPMI/DCMI on UDP/623. That stack has no business being reachable from the wider LAN. On MGMT, only my laptop and the host itself can talk to it, and the pfSense rules between SERVERS and MGMT make sure nothing on SERVERS can escalate through it.

DMZ is where the internet gets in

VLAN 300 is 203.0.113.0/24. One IP is in use. 203.0.113.10 is my cloudflared LXC.

Cloudflared holds the outbound TLS session to Cloudflare's edge. Requests to cloud.iamkay.eu, ha.iamkay.eu, and a handful of other public routes flow inbound through that tunnel, hit cloudflared, and cloudflared forwards them to the Traefik VRRP VIP at 10.99.99.29 on port 443. The pfSense rule that allows this flow is one line:

src=203.0.113.10 -> dst=10.99.99.29 tcp/443

Nothing else can cross from DMZ to SERVERS. Cloudflared cannot SSH into anything, cannot hit Vaultwarden directly, cannot touch Nextcloud's data volume. It has exactly one job.

Enterprise DMZ patterns describe a /24 subnet full of load balancers, WAFs, and reverse proxies, all serving multiple internet-facing apps. That is the right model at that scale. At homelab scale, my entire DMZ is one process and one address. The rule set can be minimal because the surface is minimal.

The rules that tie it together

pfSense is the only device that can route between VLANs. The rules I care about most:

SERVERS to MGMT is blocked with no exceptions. I verified this six different ways before trusting it.
DMZ to SERVERS allows exactly 203.0.113.10 -> 10.99.99.29 on tcp/443. Nothing else.
MGMT to anywhere is allowed, because MGMT is my hands. If someone is on MGMT they either have physical access to the switch or my laptop.
SERVERS to the internet is allowed for outbound traffic: updates, DNS forwarding, mail relay, ACME challenges.

The switch itself is a TP-Link TL-SG108E v1. Static VLAN tagging, not 802.3ad. That kept me from doing LACP with pfSense, which is a whole other story I will get to. The host runs a single active-backup bond over two 1GbE NICs. VLAN tags terminate on vmbr0.100, vmbr0.200, and vmbr0.300 sub-interfaces.

Tradeoffs I know I have

Family devices are on the untagged VLAN 1 today. That is not a decision, it is an omission. When I move them to their own VLAN, the pfSense rules need to say family devices can reach cloud.iamkay.eu and nothing else on the homelab. That work is queued and not urgent. Right now the family is one household and I trust the trust boundary.

I do not have a full L3 switch. The TL-SG108E does static VLAN tagging fine, but everything routes through pfSense. If pfSense is a single point, MGMT is the fallback. If both are down at once, I am at the chassis.

What it comes back to

Each of the three VLANs is named for the thing it does. SERVERS holds the workloads. MGMT is how I reach them when SERVERS is down. DMZ is one process talking to the internet. The structure was not something I designed up front. It came out of asking, at every layer, what happens if this network dies.

The cold-storage recovery would have gone sideways if the box between my laptop and pfSense had died during the VLAN cutover. It did not, but that near-miss is what taught me the shape of MGMT. MGMT refuses to route anywhere. DMZ routes exactly one thing. Both segments exist to answer what happens when the primary path breaks.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

I never ran ESXi in production

Kingsley Kanu — Sun, 26 Jul 2026 15:46:06 +0000

Most "why Proxmox" content in 2025-2026 is a migration story driven by Broadcom's ESXi pricing changes. The author had a working VMware stack and got priced out. I'm not that author. I evaluated both, picked Proxmox in 2024, and built on it without ever running ESXi in production. Two years in, I'd make the same call.

It reads as either incompetent or contrarian until the rest of the post lands. Here's the reasoning.

The three reasons it was the easy call

1. LXC and KVM in one host

Most workloads in this homelab are LXCs. Pi-hole, Vaultwarden, Authelia, Traefik, the monitoring stack, GitLab CE itself, all containers sharing the host kernel. A few things need full VM isolation (the NAS guest, Proxmox Backup Server, the Home Assistant OS appliance). Same hypervisor, same CLI, same web UI for both shapes of workload.

The alternative is ESXi for the VMs and a separate toolchain (containerd, Docker, Kubernetes, take your pick) for the containers. That's two backup pipelines, two HA stories, two places for config drift to surprise you at 2 AM.

pct exec 254 systemctl status authelia and qm start 189 are the same shape. New hires don't have to learn one tool for containers and a different one for VMs.

2. Proxmox Backup Server beats the free Veeam alternative

Chunk-level deduplication. Backups across guests and across time share storage. A nightly backup of all 11 LXCs and 2 VMs runs in about ten minutes and adds a few hundred MB of new chunks, because most of the content is the same as yesterday.

Cluster-scheduled. One job definition runs across every node in the cluster. No per-node cron, no manual rotation when a node moves.

Restore to a different storage class. A backup taken from local-lvm on the G7 restores onto ZFS on a G5 cluster node without conversion gymnastics.

Veeam Community Edition is the free comparison. It works. It also caps repository size, doesn't dedup at the chunk level, and lacks the cluster-aware scheduling that makes PBS feel like a built-in feature rather than a bolt-on.

When backups are the load-bearing wall of a homelab, the part that lets me say yes to family data later, the free option that compromises least is the one that wins.

3. Cluster and HA without vCenter

ESXi clustering requires vCenter Server. vCenter is its own VM, its own database, its own patching cadence, its own license. The minimum production footprint to get HA is three ESXi hosts plus one vCenter plus shared storage. The vCenter VM itself also needs to be backed up off the cluster it manages.

Proxmox does the same job from the per-node web UI. pvecm create, pvecm add from each new node, edit corosync.conf if the auto-detect picks the wrong network, done. HA failover policy is ha-manager add ct:256 --group g5-only. No separate management plane.

Three-node cluster running. A real workload migrated to a different node on the night I shut down one of the cluster members. Total observability downtime: about a minute and a half.

Honest tradeoffs

I want this post to read as opinionated, not as marketing.

Ecosystem. VMware has decades of third-party integrations across storage vendors, backup vendors, networking overlays, and security tooling. Proxmox has fewer. If you need a specific compliance-certified storage appliance or a vendor-specific tooling chain, VMware is sometimes the only choice.

Audit logging. Proxmox logs the basics. Deep audit trails with role hierarchy, per-action attestation, and tamper-evident logs are an ESXi + Aria stack feature that doesn't have a Proxmox equivalent. For SOC 2, HIPAA, or FedRAMP work this matters. For a homelab and a small team it doesn't.

Support. The Proxmox Enterprise subscription exists and is reasonable, but the support contract isn't the same as VMware's. For 24/7 white-glove production support, ESXi still wins. For the small-team case, the community forums and the wiki are usually faster anyway.

The honest summary: at small scale Proxmox wins on tooling unification, on free-tier capability, and on the speed at which you can change your mind. At enterprise scale the tradeoffs reverse.

What kept me here

I haven't reconsidered in two years because the daily friction is low. The web UI does what I want, the CLI is consistent across LXC and VM, and the backup story stays one tool. Migrating a service between nodes is two commands; when a node falls over the cluster handles it.

The homelab keeps growing. Every additional service is a new LXC or VM in the same pane of glass. The platform hasn't forced a new toolchain on me to keep up.

The right tool depends on the scale you actually operate at. For mine, it's Proxmox.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.