DEV Community: Kingsley Kanu

Where my secrets live, and what the repo gets instead

Kingsley Kanu — Thu, 15 Oct 2026 09:00:00 +0000

A repository is a very good place to lose a secret. It is copied to every clone, mirrored to every remote, kept forever, and read by tools that never ask whether a line was meant to be private. So the rule I work to is simple: the repo holds the shape of the configuration, never the values.

What goes where

Secrets live in a password manager I host myself. Nothing else is the source of truth. A database password, an API token or a private key exists in exactly one place, and everything else refers to it.

The repository carries a template instead. Each project has an example environment file with the same variable names as the real one and placeholder values that mean nothing. Anyone reading the repo can see what has to be supplied. Nobody reading it can see what was supplied.

The real values reach a machine at deploy time, through an environment file that is created on the target and never committed. The build and deploy scripts read from the environment. They do not contain a value to fall back on, because a default password in a script is a committed password with extra steps.

Three habits that do the real work

Look at what is staged before every commit. git diff --cached takes seconds, and it catches the file that was only supposed to be in the working tree for a minute.
Make the ignore file say it first. The real environment file is ignored from the first commit of a project, before it exists, so there is never a moment when adding everything picks it up.
Scan automatically as well. A secret scanner running as a pre-commit hook, gitleaks being one example, catches the cases where attention was elsewhere. It is a second net, not a replacement for the first.

When one gets committed anyway

Deleting it in the next commit does not help. The old commit still has it, every clone still has it, and any mirror may already have copied it. Once a secret has been committed it is compromised, and the fix is to rotate it at the source: issue a new one, change what uses it, revoke the old one. Cleaning the history is worth doing afterwards, but it is tidying, not remediation.

That ordering matters because the instinct goes the other way. Removing the file feels like the fix, and rotating the secret feels like admitting something happened. Rotating is the one that actually closes the exposure.

Why bother in a homelab

It is tempting to treat a private repo on my own network as safe enough. I do not, because a habit that only switches on when the stakes are high will not be there on the day they are, and the repo that is private today gets a second remote, a collaborator or a mirror tomorrow. Keeping values out of git from the start costs almost nothing. Retrofitting it later costs a rotation of everything that was ever in there.

The cluster joined fine. Its heartbeat was on the wrong network.

Kingsley Kanu — Thu, 15 Oct 2026 08:00:00 +0000

The day I formed my three-node Proxmox cluster, the create went cleanly, the first join went cleanly, and the cluster reported quorate. Every green light a person could ask for. It took a closer look at the config to notice that one node's cluster heartbeat was running over the wrong network entirely, and that nothing anywhere was going to warn me about it.

The one-flag mistake

Forming a Proxmox cluster is two commands. On the first node:

pvecm create clustername

And on each joining node:

pvecm add <address-of-first-node>

That second command took my instruction about which node to join, and quietly made its own decision about which of the joining node's addresses to register as the cluster link. My nodes are multi-homed: each has an address on the server network, where cluster traffic belongs, and another on a management network that exists for admin access. I pointed pvecm add at the first node's server-network address and assumed the joining node would present its matching interface.

It presented the management interface instead. The flag I did not pass is --link0, which pins which local address the node registers for corosync, the protocol that carries cluster membership and configuration consensus. Without it, the node picks by its own resolution logic, and what it picks is whatever your naming and routing happen to serve up, which on a multi-homed box is a coin toss you did not know was being flipped.

The result: two nodes heartbeating on the server network, one heartbeating from its management address, and a cluster that reports healthy because packets do, in fact, arrive. For now.

Why "it works" was the dangerous part

If the wrong link had failed to connect, I would have caught it in the first minute. Instead everything worked, because the management network happened to route to the others. The problem was latent, and it had teeth:

The management network is not built for cluster traffic. It is the network I take down, re-firewall, and generally treat as a maintenance surface. Corosync is the one protocol that must never flap: if a node loses heartbeat for long enough, the cluster considers it dead, and with high availability enabled, a node that loses quorum will fence itself, a deliberate hard reset to protect shared state. A firewall change on a network I believed was "just admin access" could have rebooted a production node.

That failure would have arrived months later, disconnected from its cause, on an afternoon when I touched something apparently unrelated. The bugs that pass every test and then sit waiting are the worst kind in infrastructure.

The fix: editing the cluster's own consensus file

The corosync config lives in the cluster filesystem, and you fix a wrong ring address by editing it directly, carefully:

cp /etc/pve/corosync.conf /root/corosync.conf.backup
nano /etc/pve/corosync.conf

Inside, each node has a ring0_addr. I corrected the wrong one to the node's server-network address, and, critically, incremented config_version in the totem block. That version bump is how corosync knows this is a newer config to converge on rather than a conflicting one to argue about. Skip it and nodes can reject the change or, worse, split over it.

Then restart corosync on every node, one at a time, watching quorum recover between each:

systemctl restart corosync
pvecm status

Total fix time, once understood: about ten minutes. The cluster never lost quorum during the rolling restart, and the ring has lived on the correct network since.

What I actually learned

The command that joins a node to a cluster takes a flag that pins which network the cluster's nervous system runs on. On any multi-homed node, that flag is not optional politeness; it is the difference between choosing your topology and being assigned one.

So the join I run now, every time, looks like this:

pvecm add <first-node> --link0 <this-node's-cluster-address>

And the check after any join, no matter how green the output: read the config back and confirm every ring0_addr is an address you chose.

grep ring0_addr /etc/pve/corosync.conf

Green lights tell you packets arrived. They do not tell you the packets took the road you intended. On the systems that fence themselves when the road washes out, you check the road.

Every request to my site redirected to itself

Kingsley Kanu — Tue, 13 Oct 2026 09:00:00 +0000

My home cloud sits behind a Cloudflare Tunnel, and the tunnel hands traffic to a Traefik reverse proxy. The day I wired the two together, the page would not load. The browser said it had been redirected too many times.

Nothing was down and every component answered. That made it slow to find: each piece was working, and the loop only existed in the way they were joined.

What the loop looked like

A redirect loop is a conversation where each side keeps giving the same answer. The first thing to establish is who is answering. The cheapest way to find out is to skip the browser and ask the origin directly, with a header that says which site you want:

curl -sI http://<origin>/ -H "Host: <your-hostname>"

The answer was not the page. It was a permanent redirect whose destination was the HTTPS version of the same hostname. The origin was telling the caller to go somewhere that, once it went through the tunnel again, led straight back to the same origin.

Why it looped

Traefik had a redirect from HTTP to HTTPS configured on its port 80 entrypoint. That setting applies at the entrypoint, before any router is matched. So it does not matter what a router would have done with a request: if the request arrives on port 80, it is redirected.

My tunnel route pointed at the origin over plain HTTP. The sequence went like this:

A visitor opens the site over HTTPS. Cloudflare terminates that connection.
Cloudflare sends the request down the tunnel, and cloudflared forwards it to Traefik on port 80, in plain HTTP.
Traefik answers with a redirect to the HTTPS address of the same hostname.
The browser follows it, goes back through Cloudflare, and arrives at step 2 again.

The visitor's connection was secure the whole time. The hop that mattered was the last one, between the tunnel and the proxy, and on that hop the redirect was an instruction to repeat the request forever.

What fixed it

Three changes, none of them clever:

A real certificate for the public hostname on Traefik, issued with the DNS challenge, so the proxy could serve HTTPS for that name.
The tunnel route moved to the HTTPS port, so requests never touch the redirecting entrypoint at all.
One line of resolver config on the cloudflared container. cloudflared checks the origin certificate against the hostname in the route URL. The newer Cloudflare dashboard no longer exposes the origin server name or a skip-verification option on routes, so I made the container resolve the public hostname to the proxy's internal address with an extra_hosts entry. The certificate name then matches the URL, and the traffic stays inside the house. Every public app added to that tunnel later needs the same one line.

What I check first now

Does the origin answer with a redirect? One curl -sI with a Host header, bypassing the browser and the tunnel.
Where is that redirect defined? In Traefik it can sit on the entrypoint or on a router, and the entrypoint wins.
Is the hop into the origin HTTP or HTTPS? A redirect to HTTPS is right for a browser talking to the proxy and wrong for a tunnel that already terminated TLS upstream.

Same configuration, two meanings of "HTTP". The proxy was doing its job, and so was the tunnel. They just disagreed about which end was the front door.

The boring naming decision that unlocks cluster replication

Kingsley Kanu — Tue, 13 Oct 2026 08:00:00 +0000

When I built the storage pools on my cluster nodes, the most consequential decision was not the drive, the layout, or any of the tuning options. It was the name. Both pools are called tank. Not tank1 and tank2, not a name derived from each node's hostname, not anything descriptive. The same word on every node.

Every instinct from general software engineering says this is wrong. Names should be unique. Names should tell you which thing you are looking at. Two different pools on two different machines with the same name feels like a collision waiting to happen.

Those instincts are correct for identifiers that live in one namespace. They are exactly backwards for identifiers that define equivalent slots on interchangeable machines, and a cluster's storage is the second kind.

Why the same name is the whole feature

Proxmox replication and migration move a guest's disk from a pool on one node to a pool on another. The guest's configuration names its storage once: this container's root disk lives on storage tank-local, which maps to pool tank. When the guest moves, or when its replica activates on the other node, the configuration does not change. It says tank, the receiving node has a tank, everything resolves.

Name the pools tank1 and tank2 instead and every movement between nodes becomes a translation problem. The replication has a working source and a mismatched destination. The config that says tank1 lands on a node that only knows tank2. Tools that assume the name resolves identically everywhere, which is most of them, either refuse or need per-move hand-holding. You have taken a single-command operation and given it a lookup table to maintain forever.

The rule generalizes beyond storage. Any time machines are supposed to be interchangeable, the resources that make them interchangeable must be indistinguishable. Same pool name, same dataset structure underneath it, compatible layout, same platform version. The node's own hostname is where individuality belongs; the slots a workload plugs into should be identical everywhere, like electrical sockets.

What it looks like when it works

With both pools named tank and replication scheduled, moving a container between nodes is one command with no storage arguments at all. The failover case is even better: the cluster's HA layer restarts a guest on the surviving node from the replicated copy without consulting anyone about where the disk lives. It knows where the disk lives. It lives on tank, the same as everywhere.

I tested that failover deliberately, powered a node down and watched the workload come back on its partner in about ninety seconds. That test, and the timeline of what happens in those ninety seconds, is next week's story. The point for today is that none of it works without the naming convention underneath.

Renaming later is the expensive path

If you get this wrong at creation, fixing it is not a rename in the friendly sense. A ZFS pool's name is baked into the mounting structure, the hypervisor's storage registrations, every guest config referencing it, and any replication chains already flowing. Renaming means exporting the pool, re-importing under a new name, and re-pointing everything that referenced the old one, with replication state broken and rebuilt from scratch. On a live node that is an afternoon of careful work in the best case.

Thirty seconds of deciding "every pool is called tank" at build time buys out all of that. Thirty seconds against an afternoon of careful work is not a close call.

The pool build itself, drive selection through the option-by-option rationale, is in Tuesday's post if you missed it. Together they are two halves of one idea: build each node's storage identically, and the cluster stops being three computers and starts being one system.

What the Proxmox command line does that the web UI hides

Kingsley Kanu — Thu, 08 Oct 2026 08:00:00 +0000

The Proxmox web interface is genuinely good, which is a problem, because it is good enough that you can run a cluster for a year and never learn that a second, more capable interface sits underneath it. The two command-line tools, one for containers and one for virtual machines, do everything the web UI does and a set of things it does not expose at all. The gap between them is where a lot of "you cannot do that in Proxmox" beliefs actually live.

Several things became possible the day I stopped treating the CLI as a door marked advanced users only.

Batch operations the UI makes you do one click at a time

The web UI is built around one guest at a time: select it, act on it, repeat. When you need to do the same thing to fifteen containers, that is fifteen rounds of the same clicks, and by the tenth you will make a mistake.

The command line takes a list. Restarting or reconfiguring a whole group of guests is a single loop, and a loop does not get bored on the fourteenth item. Anything you would dread doing by hand across many guests, the CLI turns into one line you can read, check, and then run once.

Configuration keys the UI never surfaces

Every guest's real configuration is a plain text file, and the CLI reads and writes it directly. The web UI shows the common subset of settings; it does not show all of them. There are behaviors, boot and startup ordering, specific device and passthrough options, particular flags, that exist as configuration keys but have no checkbox anywhere in the interface.

If you have ever read a forum answer that says "add this line to the guest config" and wondered why the setting was not in the UI, this is why. The setting was always there. The UI simply curates which settings it presents, and the CLI is the uncurated view.

Scripting, which changes what a homelab even is

This is the real unlock. A tool that takes a list and reads and writes plain text is a tool you can put in a script. Once guest management is scriptable, a class of things becomes possible that a point-and-click interface structurally cannot offer: create-from-template flows tuned exactly to your setup, scheduled maintenance actions, reproducible builds where the definition of a guest lives in a file in a git repository rather than in a series of clicks nobody wrote down.

That last shift is the important one. Clicking through the UI produces a running guest and no record of how you made it. A script produces the same guest and is itself the record. The difference shows up the day you have to rebuild, and it is the difference between remembering and re-deriving.

When to use which, honestly

None of this means abandon the web UI. For looking at things, the console, the graphs, the current state of the cluster, the UI is faster and better, and I use it constantly. For a one-off change to a single guest, clicking is genuinely quicker than typing.

The CLI earns its place the moment the work is repetitive, the moment the setting you need is not on screen, and the moment you want the action to be something you can save, review, and run again. Those three moments are exactly where the UI stops helping and, if you do not know the CLI exists, exactly where you conclude the platform cannot do the thing.

It can. It just does not always advertise it. Once you know the command line is there, the list of things you believe Proxmox cannot do gets a lot shorter.

A sub-100-dollar ZFS data tier from a used office PC

Kingsley Kanu — Fri, 18 Sep 2026 10:35:25 +0000

My cluster's newer nodes are small-form-factor office machines, the kind that companies lease for three years and then sell by the pallet. They came with one NVMe boot drive and an empty 2.5 inch SATA bay. Filling that empty bay turned each node into something with a real storage tier, for less than the price of a restaurant dinner.

The recipe is short: a used enterprise SATA SSD in the spare bay, ZFS on top of it, and a handful of pool options that people usually copy from a forum thread without knowing why. The options are the interesting part.

The drive: used enterprise, not new consumer

The drive I put in each node is an Intel S3520 class enterprise SSD. Used, off the second-hand market, for roughly the cost of a new budget consumer drive.

The case for used enterprise over new consumer at the same price is straightforward. Enterprise SATA drives have power-loss protection capacitors, which matter more than any other single feature when the filesystem on top is ZFS and the workload is VM disks. They publish honest endurance ratings, and a lightly used one typically arrives with a few percent of its rated writes consumed. A new consumer drive at the same price has none of the capacitors and a fraction of the endurance.

Before trusting any second-hand drive, read its own account of its history:

smartctl -i -A /dev/sdX | grep -E 'Model|Serial|Reallocated|Power_On_Hours|Wear'

What you want: zero reallocated sectors, wear levelling well under 80 percent used, and a power-on-hours figure consistent with the seller's story. A drive that fails any of those is a drive someone else should own.

The pool: every option earns its place

One drive, one pool. Creation looks like this:

zpool create -o ashift=12 \
  -O compression=lz4 \
  -O atime=off \
  -O xattr=sa \
  -O acltype=posixacl \
  -O dnodesize=auto \
  -O normalization=formD \
  tank /dev/disk/by-id/ata-MODEL_SERIAL

Two things before the option list. First, the pool is built on the /dev/disk/by-id/ path, never /dev/sda. Device letters are assigned by enumeration order and change when you add a disk; by-id paths are welded to the drive's serial number and survive anything short of the drive dying. Second, the pool is named tank on every node in the cluster, identically. That naming choice looks cosmetic. It is what makes replication and migration between nodes work as single commands. It deserves its own write-up, and it gets one on Thursday.

The options, and what each one buys:

ashift=12 tells ZFS the drive writes in 4K blocks. SSDs lie about their sector size for compatibility; taking the lie at face value costs write amplification forever, because ashift is fixed at creation.
compression=lz4 is close to free on any modern CPU and typically returns 1.5 to 2 times effective space on VM workloads, which are full of text, logs, and zero pages. There is no realistic workload where turning it off wins.
atime=off stops the filesystem updating a timestamp on every read. Without it, reading a file costs a write. Nothing on a VM data tier cares about access times.
xattr=sa stores extended attributes inside the inode instead of in hidden directories, one lookup instead of two. Containers touch xattrs constantly.
acltype=posixacl gives containers the ACL semantics they expect. Some distributions inside containers misbehave subtly without it, and subtle misbehaviour is the worst kind.
dnodesize=auto lets metadata-heavy datasets grow their dnodes instead of spilling attributes elsewhere. Pairs with xattr=sa.
normalization=formD makes the filesystem treat Unicode names consistently, so you cannot end up with two files whose names look identical but differ in bytes. You only hit this once, but you remember it.

Then register the pool with the hypervisor:

pvesm add zfspool tank-local --pool tank --content rootdir,images --sparse 1

The storage ID tank-local and the pool name tank are deliberately different names for different layers. --sparse 1 makes volumes thin, so a 32 GB container that holds 4 GB of data consumes 4 GB.

What this tier is for, and what it is not for

With one of these in each node, containers and VMs live on local ZFS with lz4 behind them, snapshots are instant, and scheduled replication can keep a copy of a guest's disk on the other node a few minutes stale. That replication is the foundation the cluster's failover story is built on, and both of the next two posts in this series stand on it.

What a single-drive pool does not give you is redundancy. One drive is one failure domain; ZFS will detect corruption but has no second copy to heal from. My answer to that is nightly backups to a dedicated backup server plus the cross-node replication, which together bound any loss to a small window. A mirror would be better and the bay count says no. So the redundancy lives in the backup schedule instead of in the second bay I do not have.

Total cost per node: one used enterprise SSD, somewhere between 30 and 50 dollars depending on the week. Capability added: snapshots, compression, replication, and a place for guests to live that is not the boot drive. It is the best price-to-capability ratio anywhere in my rack.

The full node build procedure, from bare hardware to cluster member, lives in my runbooks. This drive-and-pool step is section five of it, and it is the section I would keep if I had to throw away the rest.

The one SSH command Windows doesn't ship

Kingsley Kanu — Thu, 03 Sep 2026 08:00:00 +0000

Every time a fresh machine comes up on my network, the first thing I want is my laptop's key in its authorized_keys file. On Linux or macOS that is one command you have known for years:

ssh-copy-id root@192.0.2.20

On Windows, from PowerShell, that command does not exist. Not "is not on your PATH." Does not exist.

ssh-copy-id : The term 'ssh-copy-id' is not recognized as the name of a cmdlet,
function, script file, or operable program.

Windows has shipped a real OpenSSH client since Windows 10, and it is genuinely good. Here is everything in it:

scp.exe   sftp.exe   ssh.exe   ssh-add.exe   ssh-agent.exe
ssh-keygen.exe   ssh-keyscan.exe   ssh-pkcs11-helper.exe   ssh-sk-helper.exe

Nine binaries. No ssh-copy-id. The reason is that ssh-copy-id was never a binary. It is a POSIX shell script that ships alongside OpenSSH, and Microsoft's port compiles the C programs rather than carrying the shell scripts. So the omission is structural rather than an oversight, which also means it is not going to appear in a future update.

Do the same job in one line

The script does something simple: read your public key, connect, create the directory, append the key, set the permissions. You can do all of it inline.

Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh root@192.0.2.20 `
  "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

You type the password once, in your own terminal, and never again for that host.

Two Windows-specific things that trip people up in that line. The path is $env:USERPROFILE\.ssh\, because ~/.ssh is not a thing PowerShell expands the way a POSIX shell does. And the backtick is PowerShell's line continuation character, not a backslash.

If you would rather not pipe text between a Windows program and a remote shell at all, this form avoids the whole class of line-ending and encoding questions by passing the key as an argument:

$key = (Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" -Raw).Trim()
ssh root@192.0.2.20 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && printf '%s\n' '$key' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

This is the one I use, because it is explicit about the trailing newline instead of hoping the pipeline gets it right.

Or use the copy you already have

If you have Git for Windows installed, you already have the real script. Git Bash bundles the MSYS2 build of OpenSSH, which includes the shell scripts the Windows port leaves out:

$ command -v ssh-copy-id
/usr/bin/ssh-copy-id

So open Git Bash instead of PowerShell and ssh-copy-id root@192.0.2.20 works exactly as it does on Linux. Two different SSH clients live on the same machine, one of which has the command and one of which does not, which is a good thing to know before you spend twenty minutes deciding your installation is broken.

When the key lands and login still asks for a password

This is the failure that actually costs time, because everything looks correct. The key is in the file, the file is on the server, and the server still wants a password.

Almost always it is permissions. sshd runs with StrictModes on by default, and it will refuse a key file that anyone other than the owner can write to. That includes the home directory itself, not just .ssh.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod go-w ~            # the one people miss

sshd will not tell you this on the client side. It just falls through to the next authentication method, which looks identical to the key not being there. The server knows exactly what happened:

tail /var/log/auth.log
Authentication refused: bad ownership or modes for directory /root

Run ssh -v on the client and you will see it offer the key and get nothing back, which at least tells you the problem is on the far end.

The other common cause is having several keys. The client offers them in order, and if the server's MaxAuthTries limit is reached before it gets to the right one, you are refused for having too many keys rather than the wrong one. Pin it explicitly:

Host newnode
    HostName 192.0.2.20
    User root
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

IdentitiesOnly yes is the important line. Without it, the client will still offer everything in your agent regardless of what you specified.

Why I bother with any of this

The point is not saving keystrokes. It is that the password gets typed exactly once, locally, by me, on the machine it belongs to. It never goes into a note, a chat window, a script or a config file. After that first connection the credential for that host is a key on my laptop protected by the operating system, and adding a new machine to my network is a thirty second job rather than a decision about where to write a password down.

That is a small habit that compounds. Every host I have built in the last year got exactly this treatment on its first boot, and I have never had to go looking for one of their passwords, because for most of them I do not remember what it was.

A note on the addresses

Addresses in this post use the range reserved for documentation under RFC 5737.

Resources

Microsoft OpenSSH for Windows documentation - what ships and what does not
sshd_config StrictModes - why permissions silently break key auth
More homelab write-ups: iamkay.eu/blog

I moved a staging environment off the cloud in one evening

Kingsley Kanu — Mon, 31 Aug 2026 20:00:00 +0000

Assembyl is a multi-tenant application I work on. Four services out of one monorepo: an API, an admin interface, a tenant-facing web app, and a public landing page. Until recently the staging environment for all of that ran on managed cloud, with a hosted CI service doing build and test on every push, then building containers and deploying them to a serverless runtime, against a paid database tier.

It worked fine. It also billed every month for an environment nobody touches outside weekday afternoons, while I had a Proxmox cluster in my house with a self-hosted platform layer already running on it, next to my file storage and my git server. The hardware was bought. The electricity was already being spent. Staging was paying rent somewhere else for no reason I could defend.

The complication was that staging had to stay multi-tenant. Every tenant gets their own subdomain. So the job was never "deploy an app." It was "deploy a routing pattern that survives adding the twentieth tenant without twenty pieces of manual work."

What I tried first, and why it failed

The natural structure is the one everybody reaches for:

{tenant}.staging.example.com

One application, one wildcard domain *.staging.example.com, one mental model. I set up the whole path for it: internal DNS, a catch-all route on the reverse proxy, the tunnel configuration, the certificate request. All of it landed cleanly.

Then I tested from outside the network and got a TLS handshake failure.

*.staging.example.com is a two-level-deep wildcard, because the zone apex is example.com. Cloudflare's free Universal SSL covers the apex and one level below it. It does not cover a wildcard two labels deep. That needs Advanced Certificate Manager, which is a paid add-on at roughly ten dollars a month.

I was doing this migration specifically to stop paying a monthly bill. Trading one subscription for a smaller subscription is not a migration, it is a discount. So that was out.

The other option I considered seriously was dropping public access entirely and serving staging over a private network overlay, with the certificate issued through a DNS challenge and no public ingress at all. That works, and it costs nothing. It also means every person who wants to look at staging has to install a VPN client first. For an engineering-only environment that is a fine trade. For something where the point is that a colleague or a prospective customer can open a link in whatever browser they have, it breaks the use case entirely.

The fix was to flatten the namespace

Keep the wildcard one level deep by moving the tenant name into the label rather than in front of it:

{tenant}-as.example.com

The -as suffix acts as the namespace. Same *.example.com certificate that Universal SSL already issues for free, covering every tenant that will ever exist. No paid tier, no VPN requirement, no per-tenant certificate.

Two concessions came with it, and both are worth stating plainly rather than glossing.

Each tenant needs its own DNS record. Wildcard DNS records have to have the wildcard at the start of a label, so *-as.example.com is not a thing you can create. That is a per-tenant API call, which is automatable, but it is not zero.

And tenant names become visible in DNS. Anyone enumerating the zone's records can see who the tenants are. For a staging environment full of test accounts that is acceptable. For production it would not be, and production would need a different pattern.

In exchange: one certificate, one routing rule, and adding a tenant never touches the proxy configuration at all.

One routing rule for every tenant

http:
  routers:
    tenants:
      rule: "HostRegexp(`^[a-z0-9][a-z0-9-]*-as\\.example\\.com$`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencrypt
        domains:
          - main: "*.example.com"
      service: tenants-svc
      middlewares:
        - security-headers@file

  services:
    tenants-svc:
      loadBalancer:
        servers:
          - url: "http://192.0.2.80:80"
        passHostHeader: true

Every tenant name matches that regex. Every request lands on the platform proxy, which routes by Host header to the right container. passHostHeader is the load-bearing line: drop it and every tenant arrives at the application looking like the same host, which in a multi-tenant app is a genuinely bad afternoon.

Warm the certificate before anyone arrives

Issuing a wildcard certificate through a DNS challenge takes about thirty seconds. If the first request that triggers issuance is a real user's, that user gets thirty seconds of nothing, or a timeout. So trigger it yourself with a request for a name that does not exist yet:

curl -sk -H "Host: prewarm-as.example.com" \
  https://192.0.2.10/ -o /dev/null -w "HTTP %{http_code}\n"

That matches the regex, which triggers the certificate request, which issues and caches the wildcard. Every subsequent tenant deploy has zero wait. It costs one throwaway request and it is the difference between a launch that looks instant and one that looks broken.

What the migration actually looked like

Four applications on the self-hosted platform layer, each pointing at a different directory of the same monorepo, each with its own domain and its own build. All four went from "create application" to serving in about fifty minutes. The CI moved to a self-hosted runner on the same cluster, which had already replaced the hosted CI service earlier that week.

New recurring cost for the staging environment: nothing. The hardware was already there, already powered, already backed up nightly as part of everything else.

Why this holds up

The free tier of a CDN is genuinely the right scale for a staging environment. Tunnelled ingress plus free certificates handles public access with no port forwarding and no exposed origin. The one-level wildcard limit is only a problem if you insist on fighting it. Flatten the namespace and it stops existing.

A regex-matched catch-all route scales to any number of tenants in one block of configuration. The proxy is the piece you least want to be editing under time pressure, and this pattern means you never edit it after the first time.

Running the platform layer in its own container on the cluster keeps it from becoming a shared fate with everything else. It gets its own backup unit, its own resource ceiling, its own restart. When it needs attention, nothing else is affected.

What I would do differently

Use the flat pattern from the beginning instead of arriving at it after ninety minutes of debugging a TLS failure. The certificate limitation is documented; I just did not go looking until something broke.

Wire the deploy webhook before the first deploy rather than after. I spent the first stretch clicking Deploy by hand, which is fine once and irritating by the fifth time.

Add health check endpoints to the services before deploying rather than afterwards. Two of the four apps sat in an unknown health state for days purely because nothing was answering a probe, and an unknown state is worse than a red one because it is easy to ignore.

The one thing I deliberately have not moved is the production database, which is still on a managed tier. The homelab is not on battery backup yet, and a primary database is the wrong place to find out how your house handles a power cut. Staging can afford to learn that lesson. Production cannot.

A note on the addresses

Domains and addresses in this post are documentation placeholders on reserved ranges (RFC 5737 and RFC 2606). The architecture is exactly what runs; the specific names are stand-ins.

Resources

Coolify documentation - the self-hosted platform layer
Traefik HostRegexp matcher - the catch-all rule
Cloudflare Tunnel - ingress with no open ports
Let's Encrypt rate limits - pre-warming once is well inside them
More homelab write-ups: iamkay.eu/blog

Two machines answered to the same address

Kingsley Kanu — Thu, 27 Aug 2026 08:00:00 +0000

I was mid-way through bringing a new server onto the network when I picked an address for it. The next free one in the range, or so I thought. I said the number out loud while typing it, the way you do, and something about it felt familiar.

It was the password vault's address. It had been the password vault's address for two weeks. I had checked the machines on the network, seen nothing at that number, and moved on, because the vault was a container running on a different host and my check had only ever been going to find things on the host I was standing on.

That one I caught before it did anything. The fix was to move the vault to the next address up and let the new server keep the one I had already configured at the switch: stop the container, edit its address, update the reverse proxy backend and the internal DNS record, start it again. Ten minutes.

A few weeks later I did the same thing again and did not catch it, and that is the interesting half of this story.

The check that does not check

On a Proxmox cluster the obvious way to see what is running is to ask:

pct list
qm list

Both of those tell you about guests on the node you are logged into. They say nothing about the guests on the other nodes. If you have three hosts and you run this on one of them, you are looking at roughly a third of your estate and it does not look partial. It looks like a complete list, because it is a complete list of something.

A ping test is no better, and is worse in one specific way. It tells you whether something answers right now. A container that is stopped, or a machine that is powered off, still holds its claim on that address in its configuration, and will take it back the moment it starts.

So the second time, I provisioned a new container on the address my checks said was free. It was not free. A container on a different node had held it since a build I had left half-finished weeks earlier and had stopped thinking about.

What a duplicate address actually looks like

This is the part worth internalising, because the symptoms do not spell out "duplicate address" at all.

Ping worked from every node, consistently. Two machines both received the address resolution request, both answered it, and whichever reply arrived first won that particular exchange. Ping does not care which one replies, so it looked completely healthy from every angle I tested.

Then the actual service failed, and it failed differently depending on where I tested from.

From most machines, connecting to the application port returned connection refused. Those machines had cached the other container's hardware address, and that container has nothing listening on that port.

From one machine, everything worked perfectly. It happened to have cached the correct hardware address, so it reached the right container every time. That single working case cost me a good twenty minutes, because a service that works from exactly one place looks like a firewall rule or a routing problem, not an addressing problem.

And the reverse proxy, which sat on a node that had cached the wrong address, served an unbroken run of 502s while I stood on the one node where curl was returning 200.

Flushing the address cache does not fix this, which is the tell. The wrong machine is still there, still configured with that address, still answering resolution requests. Clear the cache and it repopulates, sometimes correctly, sometimes not. Anything that gets better on one machine and stays broken on another, and shuffles when you clear a cache, is an addressing conflict until proven otherwise.

The check that actually works

Every guest on a Proxmox cluster has its config file replicated to every node through the cluster filesystem. So the authoritative question is not "what is running here" but "what does the cluster believe it has handed out."

grep -rhE "ip=192\.0\.2\." \
  /etc/pve/qemu-server/ /etc/pve/lxc/ \
  /etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null \
  | grep -oE "192\.0\.2\.[0-9]+" | sort -u -t. -k4 -n

That returns every address claimed by every guest on every node, running or stopped, in numeric order. Gaps in the output are genuinely free. It takes about a second and it is the only check in this post that is actually authoritative.

Ping the candidate afterwards as a belt-and-braces step, by all means. But the config grep is the source of truth, and the ping is not, because a stopped guest is invisible to ping and still owns its number.

The general shape of the mistake

Both times, the failure was the same one: I ran a check whose scope was narrower than the question I was asking.

pct list answers "what is on this node." I was asking "what is on this cluster." Ping answers "what is awake at this address." I was asking "what is assigned to this address." Neither tool was wrong. Both gave complete, accurate answers to questions I had not meant to ask.

That is a much easier failure to walk into than a tool that gives you a wrong answer, because a wrong answer eventually contradicts something. A correctly-scoped answer to the wrong question agrees with everything, right up until it does not.

The habit I have taken from it is to say the question out loud before choosing the command, specifically the scope of it. Not "is this address free" but "is this address free anywhere in the cluster, including on things that are currently switched off." Phrased that way, pct list obviously does not answer it, and the right command is the one you reach for instead.

A note on the addresses

Addresses in this post use the range reserved for documentation under RFC 5737. The incidents and the behaviour are real; the numbers are stand-ins.

Resources

Proxmox cluster file system - why every node holds every config
RFC 826, Address Resolution Protocol - why the first reply wins
More homelab write-ups: iamkay.eu/blog

I deleted the original before checking the copy had landed

Kingsley Kanu — Mon, 24 Aug 2026 20:00:00 +0000

It was around half past midnight. I was renaming a Proxmox node, the last real step in a five hour session that had already gone well. Two commands left. I ran them in the order they appeared in my notes.

cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
rm -rf /etc/pve/nodes/oldname/

The copy printed nothing. The delete printed nothing. Both of those are normal outputs for a successful run, so I moved on to the next thing.

The copy had failed. The destination subdirectories did not exist yet, because the daemon that owns that filesystem had created the parent directory but had not finished creating the children. A copy into a path that is not there does not error loudly. It reports that it did nothing, which on a terminal at midnight is indistinguishable from reporting that it did everything.

So the second command deleted the only canonical copy of the configuration for twelve running production services.

What was actually gone

Every guest on that host was defined by a small text file: which disks it mounts, which network bridge it attaches to, how much memory it gets, which VLAN it sits on, what its boot order is. Twelve of those, covering the family file storage, the password vault, the reverse proxy, both DNS resolvers, the monitoring stack, the git server.

Not the data. The data was untouched, sitting on its volumes exactly where it had always been. What I had destroyed was the description of how any of it fits together. Which is arguably worse, because data you can point at. A container with no config is not a container, it is an unlabelled disk image.

Why nothing went down

Here is the part that saved me, and it is worth understanding properly rather than treating it as luck.

A running container or virtual machine does not consult its config file while it runs. The management layer reads that file once, at start, uses it to build the runtime, and then the running process carries on from what it already has in memory. Nothing re-reads the directory during normal operation. The management daemon only rebuilds its picture of the world from those files on a cold start.

So for as long as nothing restarted, every one of those twelve services kept serving traffic as though nothing had happened. Nextcloud carried on syncing, the vault carried on unlocking, and DNS never missed a query. My family noticed nothing, that night or ever.

That is a genuine forgiveness window, and it is a real property of how the system is built rather than a fluke. It is also the most dangerous kind of safety net, because it is invisible. Nothing tells you that you are inside it. If I had rebooted that host, or if the power had cut, I would have come back up to twelve stopped services and no record of how any of them were configured. The window was open. It was not going to announce when it closed.

The recovery

I had taken a full copy of the whole configuration tree to a plain directory on local disk about thirty minutes earlier, as the first step of the rename. Not a snapshot inside the same filesystem I was about to break, an ordinary directory somewhere else that I could read with cat when everything clever had stopped working.

cp -a /etc/pve /root/etc-pve.backup-pre-rename-20260617-001153

Recovery was copying the config files back into the correct node directory, once it existed properly, and restarting the management services. Around four minutes of work. All twelve guests reappeared in the interface still running, with the same process IDs they had held before the rename started, because they had never stopped.

That backup was the only reason this is a story about a mistake rather than a story about a rebuild.

What actually went wrong

It is tempting to file this under "be careful with rm -rf" and move on. That is not the lesson, and treating it as the lesson is how you make the same mistake again with a different command.

The real error is that I turned an operation with built-in safety into two operations without it. mv is atomic. It either moves the thing or it does not, and it will not leave you in a state where the source is gone and the destination is empty. I could not use mv here, because the daemon had already created the destination and mv refuses to move onto a non-empty directory. So I reconstructed mv by hand out of cp and rm.

The moment you do that, you have taken on a job the kernel was doing for you: checking that the first half succeeded before running the second half. I did not do that job. I just typed both halves.

The second error is that I read silence as success. cp -av is verbose, that is what the v is for, and it printed no filenames because it copied no files. I saw an absence of errors where I should have seen an absence of output. Those are different things, and at midnight they look identical.

The rule I actually wrote down

Verify the destination before destroying the source. Every destructive command needs a gate in front of it, and the gate has to check for presence, not for the absence of an error.

In practice, for this specific operation:

cp -av /etc/pve/nodes/oldname/lxc/. /etc/pve/nodes/newname/lxc/

# the gate
src=$(ls /etc/pve/nodes/oldname/lxc/ | wc -l)
dst=$(ls /etc/pve/nodes/newname/lxc/ | wc -l)
echo "source: $src  destination: $dst"
[ "$src" -eq "$dst" ] && [ "$src" -gt 0 ] || { echo "MISMATCH, stopping"; exit 1; }

Four extra lines. It would have caught this instantly, because the destination count would have been zero and the source count twelve.

There is also a simpler answer that I have adopted since, which is to not run the delete at all. Nothing required me to remove the old directory. It was a handful of small text files taking up a trivial amount of space on a machine with hundreds of gigabytes free. I deleted it because leaving it there felt messy.

That is the part that stays with me. The only genuinely destructive command in a thirty minute procedure was the one that served no purpose. It was not in service of the goal. It was housekeeping. And housekeeping is a bad reason to run rm -rf against the only copy of anything.

What I would tell someone about to do this

Take the backup to somewhere outside the system you are about to modify, and take it first, before you feel like you need it.

Put a verification gate between any copy and any delete that depends on it, and make the gate assert that files are present rather than assert that nothing errored.

Do the destructive work at the start of an evening, not the end. This was a sequencing failure, and sequencing is the first thing that degrades when you are tired and the checklist is nearly finished.

And if a step exists only because the leftovers look untidy, consider not doing it.

A note on the names

Hostnames and paths in this post use documentation placeholders. The incident, the mechanism and the recovery are exactly as they happened.

Resources

Renaming a live Proxmox node - the procedure this happened inside
Proxmox pmxcfs documentation - why the destination gets created for you
More homelab write-ups: iamkay.eu/blog

The packets left. Nothing came back.

Kingsley Kanu — Thu, 20 Aug 2026 08:00:00 +0000

I had just finished renaming a Proxmox node. Every guest was still running, the cluster was quorate, the web UI certificate had been regenerated under the new name. I ran the last check, pinged my router from the host, got four clean replies, and went to open the web UI from my laptop.

Nothing. No certificate warning, no timeout page, just a connection that hung and eventually gave up.

The containers on that host were all still reachable. Nextcloud loaded fine, so did the password vault and the dashboards. Only the host underneath them had gone dark, and only from the outside.

The symptom is the confusing part

Outbound worked perfectly. From the host I could ping anything on the network, resolve DNS, reach the internet, SSH out to another machine. From anywhere else, the host was a black hole. SSH hung, the web UI hung, ping got no reply.

That asymmetry is what sends you looking in the wrong place. Networking problems are usually symmetric. A bad route, a dead link, a wrong VLAN tag, a bridge in the wrong state, all of those break traffic in both directions. So I checked every one of them:

ip -br addr        # correct addresses on the right interfaces
ip route           # default route present and correct
bridge link        # bridge ports up, VLANs correct
ping 192.0.2.1     # replies fine

Everything healthy. Meanwhile a tcpdump on the host showed inbound SYN packets arriving from my laptop. They were reaching the machine. Nothing was going back.

At that point the answer is almost always a firewall, and I still nearly missed it, because I had not touched the firewall. I had renamed a host.

The cause

pve-firewall status

Status: enabled/running (pending changes)

That parenthetical is the whole story.

Proxmox keeps firewall configuration in two layers. There is the on-disk config, a cluster-wide cluster.fw plus a per-node host.fw living under /etc/pve/nodes/<hostname>/. And there are the actual running kernel rules, which the pve-firewall daemon generates from that config and pushes into iptables.

Rename the host and the per-node config moves to a new path along with the rest of the node's state. The daemon notices that what is on disk no longer matches what it pushed into the kernel, and it flags the drift as pending changes. What it does not do is apply them. The kernel keeps enforcing the rules it was given before the rename, and those rules scope the management access list to a node name that no longer exists.

Outbound stayed up because the default outbound policy is accept and does not depend on per-node scoping. Inbound died because the accept rule that should have matched my laptop was keyed to the old name.

The guests kept working for a completely different reason: they reach the network through the bridge, not through the host's own IP stack, so the host firewall's input chain was never in their path. That is why the box looked half alive. Half of it was.

The fix

pve-firewall restart

Under a second. Reachability came back before I had finished reading the output.

Note that it is restart, not reload. There is no reload subcommand on Proxmox 8, which is a small thing that will waste a minute if you are guessing at syntax while locked out of a box.

And you want to be on the console or an out-of-band path when you run it, because if you are somehow still connected over SSH through a rule that happens to be working, restarting the firewall is the exact operation that could drop you.

Why this bites so hard

The failure has three properties that make it worse than an ordinary outage.

It is silent. Nothing logs an error. pve-firewall status reports itself as enabled and running, which is true, and buries the important word in brackets after it. If you check services with a script that greps for running, this passes.

It is asymmetric, so it does not look like a firewall. Your instinct on "I can get out but not in" is to look at routing, NAT and the upstream, in roughly that order. A host firewall that lets everything out is not where most people start.

And it is delayed. The rename appears to succeed. Every check you run on the host itself comes back clean, because every check you run on the host itself is outbound. You only discover the problem when you go back to your own machine, which is usually after you have declared the work finished and closed the console.

What I changed

The rename procedure in my own runbook now ends with pve-firewall restart as a numbered step, not a footnote, followed by a verification that has to be run from a different machine:

# on the renamed node
pve-firewall status          # must not say "pending changes"

# from any OTHER host on the network
ping -c 2 <node-ip>
curl -sk -o /dev/null -w '%{http_code}\n' https://<node-ip>:8006/

The rule I actually took from it is broader than the command. Any procedure that changes a machine's identity should end with a check performed from somewhere that is not that machine. Everything I ran on the host told me the host was fine. The host was not fine. It just had no way to tell me, because telling me was the thing that was broken.

A note on the addresses

Addresses in this post use the ranges reserved for documentation under RFC 5737. The behaviour is real; the specific addresses are stand-ins so nothing here maps to a live target.

Resources

Proxmox VE firewall documentation - the config layers and how they are applied
Renaming a live Proxmox node - the procedure this gotcha belongs to
More homelab write-ups: iamkay.eu/blog

I renamed a live Proxmox node and twelve guests never noticed

Kingsley Kanu — Mon, 17 Aug 2026 20:00:00 +0000

The host had the wrong name. I had given it the good name, the one I wanted for the whole cluster, back when it was the only machine I owned. Then I bought two more machines, and the good name was sitting on the box least likely to survive the next two years.

So the node needed a new name, and it needed it while twelve production guests were running on it. Nextcloud, the password vault, the reverse proxy, the DNS resolvers, the monitoring stack. Things my family notices when they stop.

Renaming a Linux box is one command. Renaming a Proxmox node is not, because Proxmox keeps a per-node directory of state keyed by hostname, inside a filesystem you do not fully control.

The problem

Proxmox stores cluster configuration in /etc/pve, which looks like a normal directory and is not one. It is a FUSE mount backed by pmxcfs, a small replicated database that syncs across every node in the cluster. Under it, /etc/pve/nodes/<hostname>/ holds everything that belongs to that specific machine: the guest configs in qemu-server/ and lxc/, the per-node firewall rules, the SSL certificate.

Change the hostname and that directory path no longer matches the machine. The guest configs are still on disk, but they are sitting under a name the node no longer answers to. Proxmox will happily boot into that state and show you an empty node with no VMs on it, which is a memorable thing to see at half past midnight.

The instinct is to move the directory. That instinct is wrong, and it is wrong in a way that fails safe the first time and unsafe the second.

What I tried, and what failed

My first move was the obvious one:

mv /etc/pve/nodes/oldname /etc/pve/nodes/newname

mv: cannot move '/etc/pve/nodes/oldname' to '/etc/pve/nodes/newname': Directory not empty

The target already existed. The moment pmxcfs saw the new hostname, it created the new node directory itself, complete with empty qemu-server/ and lxc/ subdirectories. You are not moving into empty space. The daemon got there first.

Fine, so copy instead:

cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/

This is where it got interesting. Run that too early, in the seconds before pmxcfs has created the subdirectories, and it does not error in a way that stops you. It reports nothing copied and returns. If you are working through a checklist at speed, "no output" reads exactly like "done."

I then did the thing I am going to write a whole separate post about, which is that I deleted the source directory before confirming the copy had landed. Twelve guest configs, gone from the only canonical copy. I will spare you the details here; the short version is that a pre-rename backup saved me and production never actually went down.

The reason production survived is worth understanding, because it is the mechanism that makes this whole operation safe. A running LXC or QEMU guest does not read /etc/pve continuously. It reads its config once, at start, and then runs from what it already holds. pve-manager re-reads the directory on a cold start. So for as long as nothing restarts, the configs can be missing entirely and every workload keeps serving. That is a real forgiveness window, and it is the only reason a rename on a live node is a reasonable thing to attempt.

It is forgiveness, not permission. A power cut during that window and I would have had twelve running services and no idea how any of them were configured.

The fix

Here is the sequence that works, with the verification gates that stop the failure modes above. Take a backup of the whole /etc/pve tree to somewhere outside /etc/pve first. Not to a snapshot, to a plain directory on local disk you can read with cat when everything else is broken.

cp -a /etc/pve /root/etc-pve.backup-pre-rename-$(date +%Y%m%d-%H%M%S)

Then:

# 1. Set the new name in both places
sed -i 's/oldname/newname/' /etc/hostname
# keep the old name as an alias in /etc/hosts through the transition
hostnamectl set-hostname newname

# 2. WAIT. Let pmxcfs notice and build the new directory itself.
#    Do not mv. Do not mkdir. Just watch for it to appear.
until [ -d /etc/pve/nodes/newname/qemu-server ]; do sleep 1; done
ls -la /etc/pve/nodes/newname/

# 3. Copy the state across
cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
cp -av /etc/pve/nodes/oldname/lxc/.         /etc/pve/nodes/newname/lxc/
cp -av /etc/pve/nodes/oldname/host.fw       /etc/pve/nodes/newname/ 2>/dev/null

# 4. VERIFY THE COUNTS MATCH before you delete anything
ls /etc/pve/nodes/oldname/qemu-server/ | wc -l
ls /etc/pve/nodes/newname/qemu-server/ | wc -l
ls /etc/pve/nodes/oldname/lxc/ | wc -l
ls /etc/pve/nodes/newname/lxc/ | wc -l

# 5. Only when those numbers agree, and honestly you can just leave the old
#    directory in place forever, it costs a few kilobytes
rm -rf /etc/pve/nodes/oldname/

# 6. Restart the services that cache the node identity
systemctl restart pve-cluster
systemctl restart pvedaemon pveproxy pvestatd

# 7. Regenerate the web UI certificate under the new name
pvecm updatecerts -f

# 8. The step everyone misses
pve-firewall restart

That last line is the one that cost me the second half of the night. Proxmox marks the firewall as having pending changes after a hostname change and does not apply them. The running kernel rules still scope to the old node name, so the host can reach outward while nothing can reach in. It is a silent, asymmetric loss of connectivity and it looks nothing like a firewall problem. That one has its own write-up on Thursday.

Verify before you call it done:

pve-firewall status     # must NOT say "pending changes"
pvecm status            # quorum intact, node listed under the new name
qm list && pct list     # every guest present and still running

And ping the node from another machine on the network. Not from itself.

Why this works

The whole procedure comes down to one idea: pmxcfs owns that directory tree, and you are a guest in it. Every step that fights the daemon fails, and every step that waits for the daemon works.

mv fails because the daemon has already created the destination. mkdir is unnecessary for the same reason. The copy has to wait because the subdirectories appear a beat after the parent does, and a copy into a path that does not exist yet is not an error, it is just nothing. The verification gate exists because "nothing happened" and "everything happened" produce the same silence on the terminal.

The service restarts are ordered deliberately. pve-cluster owns pmxcfs itself, so it goes first and the rest of the stack picks up the new identity behind it. pvecm updatecerts -f matters because the web UI certificate carries the old hostname in its subject, and browsers do care.

What I'd do differently

I would skip step 5 entirely. Nothing about a rename requires deleting the old node directory, and it is the only command in the whole procedure that can lose you something. Leaving it costs a few kilobytes and buys you a second copy of every guest config for nothing.

I would also treat the gate in step 4 as the actual work rather than as paperwork. Every other step here either succeeds visibly or fails loudly at you. Steps 3 and 4 are the only two that can quietly do nothing, and they sit directly in front of the one command you cannot undo.

The rename took about thirty minutes. Cleaning up after my own impatience took another fifteen, and that part gets its own write-up next week. Every guest kept running throughout, which says more about how Proxmox treats a live config directory than it does about my procedure.

A note on the addresses

Hostnames and addresses in this post are documentation placeholders. Any IPs elsewhere in this series use the ranges reserved for documentation under RFC 5737 and RFC 2606. The architecture is real; the specific names are stand-ins so nothing here maps to a live target.

Resources

Proxmox pmxcfs documentation - what /etc/pve actually is
Proxmox cluster manager - pvecm reference
More homelab write-ups: iamkay.eu/blog