DEV Community: Kingsley Kanu

The one SSH command Windows doesn't ship

Kingsley Kanu — Thu, 03 Sep 2026 08:00:00 +0000

Every time a fresh machine comes up on my network, the first thing I want is my laptop's key in its authorized_keys file. On Linux or macOS that is one command you have known for years:

ssh-copy-id root@192.0.2.20

On Windows, from PowerShell, that command does not exist. Not "is not on your PATH." Does not exist.

ssh-copy-id : The term 'ssh-copy-id' is not recognized as the name of a cmdlet,
function, script file, or operable program.

Windows has shipped a real OpenSSH client since Windows 10, and it is genuinely good. Here is everything in it:

scp.exe   sftp.exe   ssh.exe   ssh-add.exe   ssh-agent.exe
ssh-keygen.exe   ssh-keyscan.exe   ssh-pkcs11-helper.exe   ssh-sk-helper.exe

Nine binaries. No ssh-copy-id. The reason is that ssh-copy-id was never a binary. It is a POSIX shell script that ships alongside OpenSSH, and Microsoft's port compiles the C programs rather than carrying the shell scripts. So the omission is structural rather than an oversight, which also means it is not going to appear in a future update.

Do the same job in one line

The script does something simple: read your public key, connect, create the directory, append the key, set the permissions. You can do all of it inline.

Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" | ssh root@192.0.2.20 `
  "mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

You type the password once, in your own terminal, and never again for that host.

Two Windows-specific things that trip people up in that line. The path is $env:USERPROFILE\.ssh\, because ~/.ssh is not a thing PowerShell expands the way a POSIX shell does. And the backtick is PowerShell's line continuation character, not a backslash.

If you would rather not pipe text between a Windows program and a remote shell at all, this form avoids the whole class of line-ending and encoding questions by passing the key as an argument:

$key = (Get-Content "$env:USERPROFILE\.ssh\id_ed25519.pub" -Raw).Trim()
ssh root@192.0.2.20 "mkdir -p ~/.ssh && chmod 700 ~/.ssh && printf '%s\n' '$key' >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"

This is the one I use, because it is explicit about the trailing newline instead of hoping the pipeline gets it right.

Or use the copy you already have

If you have Git for Windows installed, you already have the real script. Git Bash bundles the MSYS2 build of OpenSSH, which includes the shell scripts the Windows port leaves out:

$ command -v ssh-copy-id
/usr/bin/ssh-copy-id

So open Git Bash instead of PowerShell and ssh-copy-id root@192.0.2.20 works exactly as it does on Linux. Two different SSH clients live on the same machine, one of which has the command and one of which does not, which is a good thing to know before you spend twenty minutes deciding your installation is broken.

When the key lands and login still asks for a password

This is the failure that actually costs time, because everything looks correct. The key is in the file, the file is on the server, and the server still wants a password.

Almost always it is permissions. sshd runs with StrictModes on by default, and it will refuse a key file that anyone other than the owner can write to. That includes the home directory itself, not just .ssh.

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
chmod go-w ~            # the one people miss

sshd will not tell you this on the client side. It just falls through to the next authentication method, which looks identical to the key not being there. The server knows exactly what happened:

tail /var/log/auth.log
Authentication refused: bad ownership or modes for directory /root

Run ssh -v on the client and you will see it offer the key and get nothing back, which at least tells you the problem is on the far end.

The other common cause is having several keys. The client offers them in order, and if the server's MaxAuthTries limit is reached before it gets to the right one, you are refused for having too many keys rather than the wrong one. Pin it explicitly:

Host newnode
    HostName 192.0.2.20
    User root
    IdentityFile ~/.ssh/id_ed25519
    IdentitiesOnly yes

IdentitiesOnly yes is the important line. Without it, the client will still offer everything in your agent regardless of what you specified.

Why I bother with any of this

The point is not saving keystrokes. It is that the password gets typed exactly once, locally, by me, on the machine it belongs to. It never goes into a note, a chat window, a script or a config file. After that first connection the credential for that host is a key on my laptop protected by the operating system, and adding a new machine to my network is a thirty second job rather than a decision about where to write a password down.

That is a small habit that compounds. Every host I have built in the last year got exactly this treatment on its first boot, and I have never had to go looking for one of their passwords, because for most of them I do not remember what it was.

A note on the addresses

Addresses in this post use the range reserved for documentation under RFC 5737.

Resources

Microsoft OpenSSH for Windows documentation - what ships and what does not
sshd_config StrictModes - why permissions silently break key auth
More homelab write-ups: iamkay.eu/blog

I moved a staging environment off the cloud in one evening

Kingsley Kanu — Mon, 31 Aug 2026 20:00:00 +0000

Assembyl is a multi-tenant application I work on. Four services out of one monorepo: an API, an admin interface, a tenant-facing web app, and a public landing page. Until recently the staging environment for all of that ran on managed cloud, with a hosted CI service doing build and test on every push, then building containers and deploying them to a serverless runtime, against a paid database tier.

It worked fine. It also billed every month for an environment nobody touches outside weekday afternoons, while I had a Proxmox cluster in my house with a self-hosted platform layer already running on it, next to my file storage and my git server. The hardware was bought. The electricity was already being spent. Staging was paying rent somewhere else for no reason I could defend.

The complication was that staging had to stay multi-tenant. Every tenant gets their own subdomain. So the job was never "deploy an app." It was "deploy a routing pattern that survives adding the twentieth tenant without twenty pieces of manual work."

What I tried first, and why it failed

The natural structure is the one everybody reaches for:

{tenant}.staging.example.com

One application, one wildcard domain *.staging.example.com, one mental model. I set up the whole path for it: internal DNS, a catch-all route on the reverse proxy, the tunnel configuration, the certificate request. All of it landed cleanly.

Then I tested from outside the network and got a TLS handshake failure.

*.staging.example.com is a two-level-deep wildcard, because the zone apex is example.com. Cloudflare's free Universal SSL covers the apex and one level below it. It does not cover a wildcard two labels deep. That needs Advanced Certificate Manager, which is a paid add-on at roughly ten dollars a month.

I was doing this migration specifically to stop paying a monthly bill. Trading one subscription for a smaller subscription is not a migration, it is a discount. So that was out.

The other option I considered seriously was dropping public access entirely and serving staging over a private network overlay, with the certificate issued through a DNS challenge and no public ingress at all. That works, and it costs nothing. It also means every person who wants to look at staging has to install a VPN client first. For an engineering-only environment that is a fine trade. For something where the point is that a colleague or a prospective customer can open a link in whatever browser they have, it breaks the use case entirely.

The fix was to flatten the namespace

Keep the wildcard one level deep by moving the tenant name into the label rather than in front of it:

{tenant}-as.example.com

The -as suffix acts as the namespace. Same *.example.com certificate that Universal SSL already issues for free, covering every tenant that will ever exist. No paid tier, no VPN requirement, no per-tenant certificate.

Two concessions came with it, and both are worth stating plainly rather than glossing.

Each tenant needs its own DNS record. Wildcard DNS records have to have the wildcard at the start of a label, so *-as.example.com is not a thing you can create. That is a per-tenant API call, which is automatable, but it is not zero.

And tenant names become visible in DNS. Anyone enumerating the zone's records can see who the tenants are. For a staging environment full of test accounts that is acceptable. For production it would not be, and production would need a different pattern.

In exchange: one certificate, one routing rule, and adding a tenant never touches the proxy configuration at all.

One routing rule for every tenant

http:
  routers:
    tenants:
      rule: "HostRegexp(`^[a-z0-9][a-z0-9-]*-as\\.example\\.com$`)"
      entryPoints:
        - websecure
      tls:
        certResolver: letsencrypt
        domains:
          - main: "*.example.com"
      service: tenants-svc
      middlewares:
        - security-headers@file

  services:
    tenants-svc:
      loadBalancer:
        servers:
          - url: "http://192.0.2.80:80"
        passHostHeader: true

Every tenant name matches that regex. Every request lands on the platform proxy, which routes by Host header to the right container. passHostHeader is the load-bearing line: drop it and every tenant arrives at the application looking like the same host, which in a multi-tenant app is a genuinely bad afternoon.

Warm the certificate before anyone arrives

Issuing a wildcard certificate through a DNS challenge takes about thirty seconds. If the first request that triggers issuance is a real user's, that user gets thirty seconds of nothing, or a timeout. So trigger it yourself with a request for a name that does not exist yet:

curl -sk -H "Host: prewarm-as.example.com" \
  https://192.0.2.10/ -o /dev/null -w "HTTP %{http_code}\n"

That matches the regex, which triggers the certificate request, which issues and caches the wildcard. Every subsequent tenant deploy has zero wait. It costs one throwaway request and it is the difference between a launch that looks instant and one that looks broken.

What the migration actually looked like

Four applications on the self-hosted platform layer, each pointing at a different directory of the same monorepo, each with its own domain and its own build. All four went from "create application" to serving in about fifty minutes. The CI moved to a self-hosted runner on the same cluster, which had already replaced the hosted CI service earlier that week.

New recurring cost for the staging environment: nothing. The hardware was already there, already powered, already backed up nightly as part of everything else.

Why this holds up

The free tier of a CDN is genuinely the right scale for a staging environment. Tunnelled ingress plus free certificates handles public access with no port forwarding and no exposed origin. The one-level wildcard limit is only a problem if you insist on fighting it. Flatten the namespace and it stops existing.

A regex-matched catch-all route scales to any number of tenants in one block of configuration. The proxy is the piece you least want to be editing under time pressure, and this pattern means you never edit it after the first time.

Running the platform layer in its own container on the cluster keeps it from becoming a shared fate with everything else. It gets its own backup unit, its own resource ceiling, its own restart. When it needs attention, nothing else is affected.

What I would do differently

Use the flat pattern from the beginning instead of arriving at it after ninety minutes of debugging a TLS failure. The certificate limitation is documented; I just did not go looking until something broke.

Wire the deploy webhook before the first deploy rather than after. I spent the first stretch clicking Deploy by hand, which is fine once and irritating by the fifth time.

Add health check endpoints to the services before deploying rather than afterwards. Two of the four apps sat in an unknown health state for days purely because nothing was answering a probe, and an unknown state is worse than a red one because it is easy to ignore.

The one thing I deliberately have not moved is the production database, which is still on a managed tier. The homelab is not on battery backup yet, and a primary database is the wrong place to find out how your house handles a power cut. Staging can afford to learn that lesson. Production cannot.

A note on the addresses

Domains and addresses in this post are documentation placeholders on reserved ranges (RFC 5737 and RFC 2606). The architecture is exactly what runs; the specific names are stand-ins.

Resources

Coolify documentation - the self-hosted platform layer
Traefik HostRegexp matcher - the catch-all rule
Cloudflare Tunnel - ingress with no open ports
Let's Encrypt rate limits - pre-warming once is well inside them
More homelab write-ups: iamkay.eu/blog

Two machines answered to the same address

Kingsley Kanu — Thu, 27 Aug 2026 08:00:00 +0000

I was mid-way through bringing a new server onto the network when I picked an address for it. The next free one in the range, or so I thought. I said the number out loud while typing it, the way you do, and something about it felt familiar.

It was the password vault's address. It had been the password vault's address for two weeks. I had checked the machines on the network, seen nothing at that number, and moved on, because the vault was a container running on a different host and my check had only ever been going to find things on the host I was standing on.

That one I caught before it did anything. The fix was to move the vault to the next address up and let the new server keep the one I had already configured at the switch: stop the container, edit its address, update the reverse proxy backend and the internal DNS record, start it again. Ten minutes.

A few weeks later I did the same thing again and did not catch it, and that is the interesting half of this story.

The check that does not check

On a Proxmox cluster the obvious way to see what is running is to ask:

pct list
qm list

Both of those tell you about guests on the node you are logged into. They say nothing about the guests on the other nodes. If you have three hosts and you run this on one of them, you are looking at roughly a third of your estate and it does not look partial. It looks like a complete list, because it is a complete list of something.

A ping test is no better, and is worse in one specific way. It tells you whether something answers right now. A container that is stopped, or a machine that is powered off, still holds its claim on that address in its configuration, and will take it back the moment it starts.

So the second time, I provisioned a new container on the address my checks said was free. It was not free. A container on a different node had held it since a build I had left half-finished weeks earlier and had stopped thinking about.

What a duplicate address actually looks like

This is the part worth internalising, because the symptoms do not spell out "duplicate address" at all.

Ping worked from every node, consistently. Two machines both received the address resolution request, both answered it, and whichever reply arrived first won that particular exchange. Ping does not care which one replies, so it looked completely healthy from every angle I tested.

Then the actual service failed, and it failed differently depending on where I tested from.

From most machines, connecting to the application port returned connection refused. Those machines had cached the other container's hardware address, and that container has nothing listening on that port.

From one machine, everything worked perfectly. It happened to have cached the correct hardware address, so it reached the right container every time. That single working case cost me a good twenty minutes, because a service that works from exactly one place looks like a firewall rule or a routing problem, not an addressing problem.

And the reverse proxy, which sat on a node that had cached the wrong address, served an unbroken run of 502s while I stood on the one node where curl was returning 200.

Flushing the address cache does not fix this, which is the tell. The wrong machine is still there, still configured with that address, still answering resolution requests. Clear the cache and it repopulates, sometimes correctly, sometimes not. Anything that gets better on one machine and stays broken on another, and shuffles when you clear a cache, is an addressing conflict until proven otherwise.

The check that actually works

Every guest on a Proxmox cluster has its config file replicated to every node through the cluster filesystem. So the authoritative question is not "what is running here" but "what does the cluster believe it has handed out."

grep -rhE "ip=192\.0\.2\." \
  /etc/pve/qemu-server/ /etc/pve/lxc/ \
  /etc/pve/nodes/*/qemu-server/ /etc/pve/nodes/*/lxc/ 2>/dev/null \
  | grep -oE "192\.0\.2\.[0-9]+" | sort -u -t. -k4 -n

That returns every address claimed by every guest on every node, running or stopped, in numeric order. Gaps in the output are genuinely free. It takes about a second and it is the only check in this post that is actually authoritative.

Ping the candidate afterwards as a belt-and-braces step, by all means. But the config grep is the source of truth, and the ping is not, because a stopped guest is invisible to ping and still owns its number.

The general shape of the mistake

Both times, the failure was the same one: I ran a check whose scope was narrower than the question I was asking.

pct list answers "what is on this node." I was asking "what is on this cluster." Ping answers "what is awake at this address." I was asking "what is assigned to this address." Neither tool was wrong. Both gave complete, accurate answers to questions I had not meant to ask.

That is a much easier failure to walk into than a tool that gives you a wrong answer, because a wrong answer eventually contradicts something. A correctly-scoped answer to the wrong question agrees with everything, right up until it does not.

The habit I have taken from it is to say the question out loud before choosing the command, specifically the scope of it. Not "is this address free" but "is this address free anywhere in the cluster, including on things that are currently switched off." Phrased that way, pct list obviously does not answer it, and the right command is the one you reach for instead.

A note on the addresses

Addresses in this post use the range reserved for documentation under RFC 5737. The incidents and the behaviour are real; the numbers are stand-ins.

Resources

Proxmox cluster file system - why every node holds every config
RFC 826, Address Resolution Protocol - why the first reply wins
More homelab write-ups: iamkay.eu/blog

I deleted the original before checking the copy had landed

Kingsley Kanu — Mon, 24 Aug 2026 20:00:00 +0000

It was around half past midnight. I was renaming a Proxmox node, the last real step in a five hour session that had already gone well. Two commands left. I ran them in the order they appeared in my notes.

cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
rm -rf /etc/pve/nodes/oldname/

The copy printed nothing. The delete printed nothing. Both of those are normal outputs for a successful run, so I moved on to the next thing.

The copy had failed. The destination subdirectories did not exist yet, because the daemon that owns that filesystem had created the parent directory but had not finished creating the children. A copy into a path that is not there does not error loudly. It reports that it did nothing, which on a terminal at midnight is indistinguishable from reporting that it did everything.

So the second command deleted the only canonical copy of the configuration for twelve running production services.

What was actually gone

Every guest on that host was defined by a small text file: which disks it mounts, which network bridge it attaches to, how much memory it gets, which VLAN it sits on, what its boot order is. Twelve of those, covering the family file storage, the password vault, the reverse proxy, both DNS resolvers, the monitoring stack, the git server.

Not the data. The data was untouched, sitting on its volumes exactly where it had always been. What I had destroyed was the description of how any of it fits together. Which is arguably worse, because data you can point at. A container with no config is not a container, it is an unlabelled disk image.

Why nothing went down

Here is the part that saved me, and it is worth understanding properly rather than treating it as luck.

A running container or virtual machine does not consult its config file while it runs. The management layer reads that file once, at start, uses it to build the runtime, and then the running process carries on from what it already has in memory. Nothing re-reads the directory during normal operation. The management daemon only rebuilds its picture of the world from those files on a cold start.

So for as long as nothing restarted, every one of those twelve services kept serving traffic as though nothing had happened. Nextcloud carried on syncing, the vault carried on unlocking, and DNS never missed a query. My family noticed nothing, that night or ever.

That is a genuine forgiveness window, and it is a real property of how the system is built rather than a fluke. It is also the most dangerous kind of safety net, because it is invisible. Nothing tells you that you are inside it. If I had rebooted that host, or if the power had cut, I would have come back up to twelve stopped services and no record of how any of them were configured. The window was open. It was not going to announce when it closed.

The recovery

I had taken a full copy of the whole configuration tree to a plain directory on local disk about thirty minutes earlier, as the first step of the rename. Not a snapshot inside the same filesystem I was about to break, an ordinary directory somewhere else that I could read with cat when everything clever had stopped working.

cp -a /etc/pve /root/etc-pve.backup-pre-rename-20260617-001153

Recovery was copying the config files back into the correct node directory, once it existed properly, and restarting the management services. Around four minutes of work. All twelve guests reappeared in the interface still running, with the same process IDs they had held before the rename started, because they had never stopped.

That backup was the only reason this is a story about a mistake rather than a story about a rebuild.

What actually went wrong

It is tempting to file this under "be careful with rm -rf" and move on. That is not the lesson, and treating it as the lesson is how you make the same mistake again with a different command.

The real error is that I turned an operation with built-in safety into two operations without it. mv is atomic. It either moves the thing or it does not, and it will not leave you in a state where the source is gone and the destination is empty. I could not use mv here, because the daemon had already created the destination and mv refuses to move onto a non-empty directory. So I reconstructed mv by hand out of cp and rm.

The moment you do that, you have taken on a job the kernel was doing for you: checking that the first half succeeded before running the second half. I did not do that job. I just typed both halves.

The second error is that I read silence as success. cp -av is verbose, that is what the v is for, and it printed no filenames because it copied no files. I saw an absence of errors where I should have seen an absence of output. Those are different things, and at midnight they look identical.

The rule I actually wrote down

Verify the destination before destroying the source. Every destructive command needs a gate in front of it, and the gate has to check for presence, not for the absence of an error.

In practice, for this specific operation:

cp -av /etc/pve/nodes/oldname/lxc/. /etc/pve/nodes/newname/lxc/

# the gate
src=$(ls /etc/pve/nodes/oldname/lxc/ | wc -l)
dst=$(ls /etc/pve/nodes/newname/lxc/ | wc -l)
echo "source: $src  destination: $dst"
[ "$src" -eq "$dst" ] && [ "$src" -gt 0 ] || { echo "MISMATCH, stopping"; exit 1; }

Four extra lines. It would have caught this instantly, because the destination count would have been zero and the source count twelve.

There is also a simpler answer that I have adopted since, which is to not run the delete at all. Nothing required me to remove the old directory. It was a handful of small text files taking up a trivial amount of space on a machine with hundreds of gigabytes free. I deleted it because leaving it there felt messy.

That is the part that stays with me. The only genuinely destructive command in a thirty minute procedure was the one that served no purpose. It was not in service of the goal. It was housekeeping. And housekeeping is a bad reason to run rm -rf against the only copy of anything.

What I would tell someone about to do this

Take the backup to somewhere outside the system you are about to modify, and take it first, before you feel like you need it.

Put a verification gate between any copy and any delete that depends on it, and make the gate assert that files are present rather than assert that nothing errored.

Do the destructive work at the start of an evening, not the end. This was a sequencing failure, and sequencing is the first thing that degrades when you are tired and the checklist is nearly finished.

And if a step exists only because the leftovers look untidy, consider not doing it.

A note on the names

Hostnames and paths in this post use documentation placeholders. The incident, the mechanism and the recovery are exactly as they happened.

Resources

Renaming a live Proxmox node - the procedure this happened inside
Proxmox pmxcfs documentation - why the destination gets created for you
More homelab write-ups: iamkay.eu/blog

The packets left. Nothing came back.

Kingsley Kanu — Thu, 20 Aug 2026 08:00:00 +0000

I had just finished renaming a Proxmox node. Every guest was still running, the cluster was quorate, the web UI certificate had been regenerated under the new name. I ran the last check, pinged my router from the host, got four clean replies, and went to open the web UI from my laptop.

Nothing. No certificate warning, no timeout page, just a connection that hung and eventually gave up.

The containers on that host were all still reachable. Nextcloud loaded fine, so did the password vault and the dashboards. Only the host underneath them had gone dark, and only from the outside.

The symptom is the confusing part

Outbound worked perfectly. From the host I could ping anything on the network, resolve DNS, reach the internet, SSH out to another machine. From anywhere else, the host was a black hole. SSH hung, the web UI hung, ping got no reply.

That asymmetry is what sends you looking in the wrong place. Networking problems are usually symmetric. A bad route, a dead link, a wrong VLAN tag, a bridge in the wrong state, all of those break traffic in both directions. So I checked every one of them:

ip -br addr        # correct addresses on the right interfaces
ip route           # default route present and correct
bridge link        # bridge ports up, VLANs correct
ping 192.0.2.1     # replies fine

Everything healthy. Meanwhile a tcpdump on the host showed inbound SYN packets arriving from my laptop. They were reaching the machine. Nothing was going back.

At that point the answer is almost always a firewall, and I still nearly missed it, because I had not touched the firewall. I had renamed a host.

The cause

pve-firewall status

Status: enabled/running (pending changes)

That parenthetical is the whole story.

Proxmox keeps firewall configuration in two layers. There is the on-disk config, a cluster-wide cluster.fw plus a per-node host.fw living under /etc/pve/nodes/<hostname>/. And there are the actual running kernel rules, which the pve-firewall daemon generates from that config and pushes into iptables.

Rename the host and the per-node config moves to a new path along with the rest of the node's state. The daemon notices that what is on disk no longer matches what it pushed into the kernel, and it flags the drift as pending changes. What it does not do is apply them. The kernel keeps enforcing the rules it was given before the rename, and those rules scope the management access list to a node name that no longer exists.

Outbound stayed up because the default outbound policy is accept and does not depend on per-node scoping. Inbound died because the accept rule that should have matched my laptop was keyed to the old name.

The guests kept working for a completely different reason: they reach the network through the bridge, not through the host's own IP stack, so the host firewall's input chain was never in their path. That is why the box looked half alive. Half of it was.

The fix

pve-firewall restart

Under a second. Reachability came back before I had finished reading the output.

Note that it is restart, not reload. There is no reload subcommand on Proxmox 8, which is a small thing that will waste a minute if you are guessing at syntax while locked out of a box.

And you want to be on the console or an out-of-band path when you run it, because if you are somehow still connected over SSH through a rule that happens to be working, restarting the firewall is the exact operation that could drop you.

Why this bites so hard

The failure has three properties that make it worse than an ordinary outage.

It is silent. Nothing logs an error. pve-firewall status reports itself as enabled and running, which is true, and buries the important word in brackets after it. If you check services with a script that greps for running, this passes.

It is asymmetric, so it does not look like a firewall. Your instinct on "I can get out but not in" is to look at routing, NAT and the upstream, in roughly that order. A host firewall that lets everything out is not where most people start.

And it is delayed. The rename appears to succeed. Every check you run on the host itself comes back clean, because every check you run on the host itself is outbound. You only discover the problem when you go back to your own machine, which is usually after you have declared the work finished and closed the console.

What I changed

The rename procedure in my own runbook now ends with pve-firewall restart as a numbered step, not a footnote, followed by a verification that has to be run from a different machine:

# on the renamed node
pve-firewall status          # must not say "pending changes"

# from any OTHER host on the network
ping -c 2 <node-ip>
curl -sk -o /dev/null -w '%{http_code}\n' https://<node-ip>:8006/

The rule I actually took from it is broader than the command. Any procedure that changes a machine's identity should end with a check performed from somewhere that is not that machine. Everything I ran on the host told me the host was fine. The host was not fine. It just had no way to tell me, because telling me was the thing that was broken.

A note on the addresses

Addresses in this post use the ranges reserved for documentation under RFC 5737. The behaviour is real; the specific addresses are stand-ins so nothing here maps to a live target.

Resources

Proxmox VE firewall documentation - the config layers and how they are applied
Renaming a live Proxmox node - the procedure this gotcha belongs to
More homelab write-ups: iamkay.eu/blog

I renamed a live Proxmox node and twelve guests never noticed

Kingsley Kanu — Mon, 17 Aug 2026 20:00:00 +0000

The host had the wrong name. I had given it the good name, the one I wanted for the whole cluster, back when it was the only machine I owned. Then I bought two more machines, and the good name was sitting on the box least likely to survive the next two years.

So the node needed a new name, and it needed it while twelve production guests were running on it. Nextcloud, the password vault, the reverse proxy, the DNS resolvers, the monitoring stack. Things my family notices when they stop.

Renaming a Linux box is one command. Renaming a Proxmox node is not, because Proxmox keeps a per-node directory of state keyed by hostname, inside a filesystem you do not fully control.

The problem

Proxmox stores cluster configuration in /etc/pve, which looks like a normal directory and is not one. It is a FUSE mount backed by pmxcfs, a small replicated database that syncs across every node in the cluster. Under it, /etc/pve/nodes/<hostname>/ holds everything that belongs to that specific machine: the guest configs in qemu-server/ and lxc/, the per-node firewall rules, the SSL certificate.

Change the hostname and that directory path no longer matches the machine. The guest configs are still on disk, but they are sitting under a name the node no longer answers to. Proxmox will happily boot into that state and show you an empty node with no VMs on it, which is a memorable thing to see at half past midnight.

The instinct is to move the directory. That instinct is wrong, and it is wrong in a way that fails safe the first time and unsafe the second.

What I tried, and what failed

My first move was the obvious one:

mv /etc/pve/nodes/oldname /etc/pve/nodes/newname

mv: cannot move '/etc/pve/nodes/oldname' to '/etc/pve/nodes/newname': Directory not empty

The target already existed. The moment pmxcfs saw the new hostname, it created the new node directory itself, complete with empty qemu-server/ and lxc/ subdirectories. You are not moving into empty space. The daemon got there first.

Fine, so copy instead:

cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/

This is where it got interesting. Run that too early, in the seconds before pmxcfs has created the subdirectories, and it does not error in a way that stops you. It reports nothing copied and returns. If you are working through a checklist at speed, "no output" reads exactly like "done."

I then did the thing I am going to write a whole separate post about, which is that I deleted the source directory before confirming the copy had landed. Twelve guest configs, gone from the only canonical copy. I will spare you the details here; the short version is that a pre-rename backup saved me and production never actually went down.

The reason production survived is worth understanding, because it is the mechanism that makes this whole operation safe. A running LXC or QEMU guest does not read /etc/pve continuously. It reads its config once, at start, and then runs from what it already holds. pve-manager re-reads the directory on a cold start. So for as long as nothing restarts, the configs can be missing entirely and every workload keeps serving. That is a real forgiveness window, and it is the only reason a rename on a live node is a reasonable thing to attempt.

It is forgiveness, not permission. A power cut during that window and I would have had twelve running services and no idea how any of them were configured.

The fix

Here is the sequence that works, with the verification gates that stop the failure modes above. Take a backup of the whole /etc/pve tree to somewhere outside /etc/pve first. Not to a snapshot, to a plain directory on local disk you can read with cat when everything else is broken.

cp -a /etc/pve /root/etc-pve.backup-pre-rename-$(date +%Y%m%d-%H%M%S)

Then:

# 1. Set the new name in both places
sed -i 's/oldname/newname/' /etc/hostname
# keep the old name as an alias in /etc/hosts through the transition
hostnamectl set-hostname newname

# 2. WAIT. Let pmxcfs notice and build the new directory itself.
#    Do not mv. Do not mkdir. Just watch for it to appear.
until [ -d /etc/pve/nodes/newname/qemu-server ]; do sleep 1; done
ls -la /etc/pve/nodes/newname/

# 3. Copy the state across
cp -av /etc/pve/nodes/oldname/qemu-server/. /etc/pve/nodes/newname/qemu-server/
cp -av /etc/pve/nodes/oldname/lxc/.         /etc/pve/nodes/newname/lxc/
cp -av /etc/pve/nodes/oldname/host.fw       /etc/pve/nodes/newname/ 2>/dev/null

# 4. VERIFY THE COUNTS MATCH before you delete anything
ls /etc/pve/nodes/oldname/qemu-server/ | wc -l
ls /etc/pve/nodes/newname/qemu-server/ | wc -l
ls /etc/pve/nodes/oldname/lxc/ | wc -l
ls /etc/pve/nodes/newname/lxc/ | wc -l

# 5. Only when those numbers agree, and honestly you can just leave the old
#    directory in place forever, it costs a few kilobytes
rm -rf /etc/pve/nodes/oldname/

# 6. Restart the services that cache the node identity
systemctl restart pve-cluster
systemctl restart pvedaemon pveproxy pvestatd

# 7. Regenerate the web UI certificate under the new name
pvecm updatecerts -f

# 8. The step everyone misses
pve-firewall restart

That last line is the one that cost me the second half of the night. Proxmox marks the firewall as having pending changes after a hostname change and does not apply them. The running kernel rules still scope to the old node name, so the host can reach outward while nothing can reach in. It is a silent, asymmetric loss of connectivity and it looks nothing like a firewall problem. That one has its own write-up on Thursday.

Verify before you call it done:

pve-firewall status     # must NOT say "pending changes"
pvecm status            # quorum intact, node listed under the new name
qm list && pct list     # every guest present and still running

And ping the node from another machine on the network. Not from itself.

Why this works

The whole procedure comes down to one idea: pmxcfs owns that directory tree, and you are a guest in it. Every step that fights the daemon fails, and every step that waits for the daemon works.

mv fails because the daemon has already created the destination. mkdir is unnecessary for the same reason. The copy has to wait because the subdirectories appear a beat after the parent does, and a copy into a path that does not exist yet is not an error, it is just nothing. The verification gate exists because "nothing happened" and "everything happened" produce the same silence on the terminal.

The service restarts are ordered deliberately. pve-cluster owns pmxcfs itself, so it goes first and the rest of the stack picks up the new identity behind it. pvecm updatecerts -f matters because the web UI certificate carries the old hostname in its subject, and browsers do care.

What I'd do differently

I would skip step 5 entirely. Nothing about a rename requires deleting the old node directory, and it is the only command in the whole procedure that can lose you something. Leaving it costs a few kilobytes and buys you a second copy of every guest config for nothing.

I would also treat the gate in step 4 as the actual work rather than as paperwork. Every other step here either succeeds visibly or fails loudly at you. Steps 3 and 4 are the only two that can quietly do nothing, and they sit directly in front of the one command you cannot undo.

The rename took about thirty minutes. Cleaning up after my own impatience took another fifteen, and that part gets its own write-up next week. Every guest kept running throughout, which says more about how Proxmox treats a live config directory than it does about my procedure.

A note on the addresses

Hostnames and addresses in this post are documentation placeholders. Any IPs elsewhere in this series use the ranges reserved for documentation under RFC 5737 and RFC 2606. The architecture is real; the specific names are stand-ins so nothing here maps to a live target.

Resources

Proxmox pmxcfs documentation - what /etc/pve actually is
Proxmox cluster manager - pvecm reference
More homelab write-ups: iamkay.eu/blog

The runner that was busy on someone else's job

Kingsley Kanu — Thu, 13 Aug 2026 08:00:00 +0000

My CI pipeline sat pending for ten minutes. The runner was healthy. So was every other pipeline in the queue. The default setting was the bug.

Healthy and stuck at the same time

GitLab Runner ships with concurrent = 1 in /etc/gitlab-runner/config.toml. That one line quietly makes every pipeline run single-file across every project sharing that runner. Not per project. Across all of them. So a chatty repo pushing every few minutes can starve everything else, and the symptom is the least helpful one possible: your job sits pending while gitlab-runner status cheerfully reports "healthy."

Healthy and stuck at the same time is the tell. The runner is not broken. It is busy, on something that is not yours.

The diagnosis

The check that confirmed it runs inside the runner's container:

docker ps --filter name=runner-
docker inspect <build-container> --format '{{.Config.Env}}'

Read CI_PROJECT_PATH and CI_PIPELINE_ID off the running build container and you learn exactly which project is holding the single lane. In my case it was a different repo entirely, mid-build, with mine queued politely behind it.

The fix and the trade

The fix is one number: bump to concurrent = 2, or higher, watching RAM, because every extra lane is another set of job containers running at once. Back up the config first, restart the runner, and the pending pipelines unblock within about thirty seconds.

The broader lesson is about defaults. concurrent = 1 is tuned for one project per runner, which is the simplest possible case and rarely the real one. The moment your runner serves several repos, that default stops protecting you and starts throttling you. "Healthy but not moving" is the signature of a default you have outgrown, not a component that has failed.

The hole I never had to open

Kingsley Kanu — Tue, 11 Aug 2026 08:00:00 +0000

My family cloud is reachable from anywhere on the internet, and I never opened a single firewall port to do it.

The hole I did not want

For years the default answer to "how do I reach my home server from outside" was port forwarding: poke a hole in your router, expose a service to the whole internet, and hope your patching keeps up with the people scanning for it. I did not want that hole. So there is not one.

How the tunnel replaces it

Instead I run cloudflared, Cloudflare's tunnel agent, in a small container on an isolated DMZ segment. It makes an outbound TLS connection to Cloudflare's edge and holds it open. When someone requests cloud.iamkay.eu, the request arrives at Cloudflare, travels back down that already-open tunnel to cloudflared, and cloudflared hands it to my internal reverse proxy, which routes it to Nextcloud. The connection is always outbound from my side. My router's inbound rules never change, because the answer to "who is allowed in" stays nobody.

The full path is worth spelling out: phone on mobile data, to Cloudflare's edge, down the tunnel to cloudflared, to the Traefik virtual IP, to Nextcloud, to the NAS behind it. No port forward anywhere in that chain.

The blast radius is one line

The part I like most is how small the exposure is. cloudflared lives on its own DMZ VLAN, and the firewall rule letting it reach the internal network is exactly one line: source 203.0.113.10, destination the proxy virtual IP at 10.99.99.29, port 443, nothing else. If cloudflared were ever compromised, it cannot SSH anywhere, cannot reach the vault, cannot touch Nextcloud's storage directly. It has one job and the firewall enforces exactly that job.

It is free for personal use, and you get Cloudflare's DDoS protection and WAF in front of your services as a bonus. If you are still port forwarding self-hosted services in 2026, this is the switch to make.

A note on the addresses

The IP addresses and hostnames in this post are documentation placeholders on reserved ranges, not my real ones. The public route cloud.iamkay.eu is real because it is a destination I want people to reach; the internal IPs are stand-ins on purpose. The architecture is exactly how I run it. Nothing here points at a live target.

The failover that fooled me

Kingsley Kanu — Thu, 06 Aug 2026 17:23:37 +0000

My reverse proxy had VRRP failover. I killed the process directly to test it, and the traffic black-holed anyway.

The setup that looked complete

I run a Traefik pair with keepalived holding a shared virtual IP between them. The idea is simple and standard: if the active node dies, the IP moves to the standby and traffic keeps flowing. And it does, if the whole node dies. Pull the power on the master and the VIP is on the backup in a couple of seconds.

The gap is what happens when the box lives but the application does not.

Node up, app down, VIP stuck

keepalived on its own watches only the VRRP heartbeat between the two peers. That heartbeat proves the box is alive and on the network. It says nothing about whether the application on that box is actually serving. So when I killed the Traefik process but left the machine running, the heartbeat kept ticking, keepalived saw a healthy master, and the virtual IP sat happily on a node that was no longer answering a single connection. The failover I thought I had covered exactly the failure that is least likely (a whole machine dying) and missed the one that is most likely (a process crashing).

The five lines that fix it

The fix is a health check that keepalived runs on a timer and folds into its own priority:

vrrp_script chk_traefik {
  script "curl -sf http://localhost/ping || exit 1"
  interval 2
  fall 2
  rise 3
}

Reference it from the vrrp_instance track_script block, and now keepalived lowers its own priority the moment the app stops answering, which hands the VIP to the standby. The same pattern works for nginx, haproxy, or any single process sitting behind a virtual IP.

What it comes back to

The heartbeat proves the box is up. The track_script proves the app is up. If your VIP fronts an application, you need both, and the only reliable way to know you have both is to test by killing the process, not the machine. The machine-death test passes on a broken config. The process-death test is the one that tells the truth.

My passwords no longer leave my house

Kingsley Kanu — Thu, 06 Aug 2026 17:22:46 +0000

My password vault runs on a box in my house now. Every credential I have syncs to my phone and my browser the same way it did when I paid Bitwarden for it, except the encrypted database lives on hardware I can walk over and touch.

That sentence is either reassuring or terrifying depending on how you feel about being your own backup department, and I want to be honest about both.

What it actually is

Vaultwarden is a reimplementation of the Bitwarden server in Rust. It speaks the same API as the official server, which is the whole point: the official Bitwarden browser extension, the mobile app, and the desktop client all work against it unchanged. You point them at your own server URL instead of bitwarden.com and they cannot tell the difference. The clients stay official and audited. Only the server is swapped.

The reason to run Vaultwarden instead of the official self-hosted Bitwarden server is footprint. The official server is a stack of containers and wants a real chunk of RAM. Vaultwarden is a single process with a SQLite database, sitting at around 50 MB resident. It runs in a Docker container inside an LXC on my cluster and I mostly forget it exists. It lives behind Traefik, which terminates TLS, so the clients reach it at https://vault.hm.example.com with a proper certificate.

The part that is not optional

Here is where self-hosting a vault stops being a fun weekend project and becomes a responsibility. That SQLite file, db.sqlite3, is the vault. Not a cache of it, not a copy, the actual encrypted store of every password I have. If I lose it and have no backup, I have locked myself out of my own life. Bitwarden's hosted service handles this with geo-redundant storage across regions. When you self-host, that is now your job.

So the backup discipline came before I trusted a single real password to it. Proxmox Backup Server takes a daily snapshot of the whole container. That covers hardware failure, a bad upgrade, a fat-fingered delete. It does not cover the house burning down, and a backup that only exists in the same building as the original is one accident away from not existing. The next step, and the one that actually brings this to parity with a hosted service, is an encrypted off-site copy. I am pushing the backup to Backblaze B2 through rclone with client-side encryption, so the copy that leaves the house is already ciphertext before it hits the wire and I hold the only key.

Why it is not behind my SSO

I run Authelia in front of most of my internal services, and Vaultwarden is deliberately not one of them. The Bitwarden browser extension and mobile app talk to the server's API directly on every sync. A forward-auth proxy answers unauthenticated requests with an HTML login page, which the extension has no idea what to do with, so it just fails to sync.

Vaultwarden does not need Authelia in front of it anyway. The vault is end-to-end encrypted with a master password the server never sees, admin registration is disabled, and Bitwarden's own brute-force throttle is on. Wrapping it in a second login would break the clients to protect data that is already encrypted at rest with a key I never upload.

The tradeoff, stated plainly

I traded roughly $40 a year and Bitwarden's geo-redundant infrastructure for full control of where my most sensitive data lives, at the cost of owning the backup risk myself. That trade is only worth it if the backup discipline is real. A self-hosted vault with no off-site copy is not cheaper than Bitwarden, it is just riskier and free, which is a bad deal dressed up as a good one.

Paired with daily PBS snapshots and an encrypted off-site copy, I am comfortable calling it parity. The vault is mine, the clients are the same ones I already trusted, and the only thing I added was the obligation to take backups seriously.

A note on the addresses

The internal hostname in this post, vault.hm.example.com, is a documentation placeholder on a reserved example domain, not the real name on my network. Any IP addresses elsewhere in this series use the ranges set aside for documentation. The architecture is real. The specific names and addresses are stand-ins on purpose, so nothing here maps to a live target.

The DL380 had been off for fifteen months

Kingsley Kanu — Mon, 03 Aug 2026 09:53:44 +0000

I powered down an HP DL380 G7 in early 2025 and did not touch it for fifteen months. It sat in a closet through a job change. On 27 May 2026 I went back to it.

Would the PSUs hold. Would the disks spin clean. Would Proxmox boot. Would I remember the passwords.

First boot

Front panel button. Fans spun up. POST stopped on a memory training failure: one DIMM in CPU socket 1, "uncorrectable error during training."

Almost opened the chassis to reseat. Waited ten minutes first. A forum post had warned me cold-stored hardware needs a few minutes to settle. Power cycled. Clean on the second attempt. No memory complaints in any boot since.

Wait through the first symptom once before you touch anything.

Proxmox came up easily

F11 at POST. Picked the boot SSD on the P410i. Proxmox came up on kernel 6.5.13-3-pve. Root password I still remembered, typed at the USB keyboard plugged into the back of the chassis.

First job at that console: move the host onto the current network. The VLAN and address from 2024 were gone. Edited /etc/network/interfaces over vi, ran ifreload -a, watched the new bridge come up, walked back to the laptop. SSH worked first try.

From there everything ran over SSH. qm list returned twenty-two VMs and one LXC. None running. Past-me had set onboot=0 on every guest before the shutdown, which is a discipline I have not always had.

Started the NAS (qm start 189), let OpenMediaVault bring up its NFS exports, checked cat /proc/mdstat:

md0 : active raid5 sdd[2] sdc[1] sdb[0]
      838860288 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid5 sdh[3] sdg[2] sdf[1] sde[0]
      1431655936 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]

Both arrays clean. The btrfs filesystems on top were readable. Fifteen months of cold did nothing to the storage layer.

iLO was the hard half.

The iLO problem

I wanted remote console access. The DL380 G7 has iLO 3, HP's out-of-band management processor from 2010. The final firmware shipped in December 2020, version 1.94 (HPE iLO firmware list).

iLO 3 v1.94 only negotiates TLS 1.0. Chrome returned ERR_SSL_VERSION_OR_CIPHER_MISMATCH with no override. Edge same answer. Firefox connected after I set security.tls.version.min = 1 in about:config. Fresh Firefox profile, dedicated to this BMC and nothing else.

The TLS workaround I had. The bigger problem was the iLO IP I had configured in 2024 was on a VLAN that no longer existed in 2026. Admin password I still remembered. I just had no route to the login page. The Proxmox host had a current address, but iLO sat on a dedicated NIC behind one that nothing on the current network could speak to.

In-band IPMI saved me a chassis open.

In-band IPMI

iLO speaks IPMI 2.0 over a KCS (Keyboard Controller Style) channel that the host OS sees as /dev/ipmi0. Root on the host gives you admin access to the BMC. No BMC password needed; you already proved you have the keys to Linux.

Debian 12 needed ipmitool. The host had no internet (network was still half-rebuilt), so I sideloaded the .deb stack from the laptop: ipmitool 1.8.19-4+deb12u2 plus freeipmi-common 1.6.10-1 and libfreeipmi17 1.6.10-1+b1, all SHA256-verified. Then:

$ ipmitool user list 1
ID  Name             Callin  Link Auth   IPMI Msg   Channel Priv Limit
1                    true    false       false      Unknown (0x00)
2   kay              true    true        true       ADMINISTRATOR

Admin in slot 2 (slot 1 is the IPMI-spec empty placeholder). Always list before you write.

Rotated the admin password. The recorded one still worked. Cheap insurance.

$ ipmitool user set password 2 'a-new-strong-password'
$ ipmitool channel setaccess 1 2 callin=on ipmi=on link=on privilege=4

The actual rescue was the IP. LAN on iLO 3 is channel 2, not 1. Channel 1 errors out, and anything you read online about "channel 1 LAN" is iLO 4 or non-HP.

$ ipmitool lan set 2 ipsrc static
$ ipmitool lan set 2 ipaddr 10.99.99.3
$ ipmitool lan set 2 netmask 255.255.255.0
$ ipmitool lan set 2 defgw ipaddr 10.99.99.1
$ ipmitool mc reset cold
Sent cold reset command to MC

The cold reset is where I burned forty minutes the first time. iLO 3 has gratuitous ARP disabled. Changing the IP does not make the BMC announce its new binding, and the upstream switch keeps the stale ARP entry for the old IP. ipmitool mc reset cold power-cycles the BMC's network stack, which is what forces the announce. Without it you watch pings drop, certain one of your lan set commands was rejected, when in fact all four were accepted just fine.

One more iLO 3 gotcha. The BMC does not log IPMI admin operations to the SEL that ipmitool sel list returns. The audit trail lives in HP's Integrated Management Log, reachable only through the web UI, RIBCL, or hponcfg, and RIBCL needs the same TLS handshake modern OpenSSL refuses. So you do recovery work and there is no on-host record. Keep your own log.

What it taught me

iLO was harder than Proxmox. That ratio will hold for any pre-2014 enterprise server. Hypervisor and storage are still maintained. The BMC is a frozen artefact of whatever firmware shipped before the vendor stopped caring.

Treat it like that. BMC on its own VLAN, no internet path. Credential on paper outside the BMC; recovery is in-band IPMI, which needs the host up. Cold-start drill once a year. Starting from scratch today I would buy a newer server.

The box came back. Data intact. Three sessions over four days. Fifteen months of cold did nothing irreversible.

Resources

HPE iLO firmware history (Wikipedia)
ipmitool upstream
Where this fits: the DL380 G7 stays as temporary quorum until a 3-node EliteDesk G5 plus WRX90 Threadripper Pro cluster takes over. More on that build in a future post.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.

The eight login forms I deleted

Kingsley Kanu — Mon, 03 Aug 2026 09:52:16 +0000

For a while every self-hosted service in my homelab had its own login form. Eight services, eight password prompts, eight places a weak password could hide. That is not security, it is friction pretending to be security.

Authelia fixed it in about two hours. This is what it does, why I chose it over the heavier option, and the one class of service I deliberately left outside it.

How it works

Authelia sits behind Traefik as a ForwardAuth middleware. Every request to an internal service gets checked against Authelia first. One login sets a cookie scoped to my internal domain, and every service under that domain trusts the cookie. Log in once, you are through to all of them. TOTP is enforced for anything in the admin group, so the sensitive services always ask for a second factor.

The internal domain here is hm.example.com, a documentation placeholder. The real one is a private split-horizon domain that only resolves inside the house. The shape is what matters: one parent domain, a wildcard cookie, and every service living as a subdomain under it.

The default that matters

The design decision worth copying is the default. Authelia's access control starts at deny, and every service I want reachable is an explicit allow rule. Adding a service means adding a line. Forgetting to add a line means the service is locked, not open. That is the failure mode I want. A system that fails closed is one you can extend without lying awake wondering what you left exposed.

Why not Keycloak

People reach for Keycloak here, and for an organisation with real identity needs it is the right tool: federation, user self-service, a full admin console. For five users it is a lot of moving parts to run and patch. Authelia is a single Go binary with a file-based user database, argon2id password hashing, and a SQLite session store. Sessions expire after five minutes idle, one hour hard cap, with a one-month "remember me" for trusted devices. It does less, and for this scale that is the point.

The service I left outside

Forward-auth returns HTML login pages. That is fine for a browser, and wrong for a sync client. The Nextcloud mobile app, CalDAV, CardDAV, and git over SSH all speak their own auth on every call and have no idea what to do with a redirect to an HTML login form. Put them behind forward-auth and they fail silently.

So those services stay outside Authelia and lean on their own authentication: Nextcloud's brute-force throttle and app passwords, git's SSH keys. The rule I follow is to match the auth pattern to what the client can actually handle, rather than wrapping everything in the same layer because it feels more secure.

What it comes back to

Single sign-on for a small system is a one-day project, not a quarter-long identity programme. One gatekeeper, a deny-by-default policy, TOTP on the admin group, and a short list of services that stay outside because their clients cannot speak the protocol. That is enough to stop running eight separate password forms, which was the whole point.

Originally published on iamkay.eu, where I write about running production infrastructure: the failures, the fixes, and the tradeoffs.