Splitting one self-hosted cluster into two, without losing a byte

For a while I ran everything on one small orchestrator: my company's tools and my personal side projects, mixed together on shared nodes, one git repo, one object storage account. It worked, but it was the wrong shape. Personal experiments could compete with production for memory, one repo held two very different blast radii, and the bill was a single lump.

So I split it. Personal services moved to their own git repo, their own cluster, their own nodes, and their own object storage project. This is part one of three: the plan and the cutover. Part two is the bug hunt that followed, and part three is the upgrade that broke everything three separate ways.

The setup

The orchestrator is orca, a small single-binary container orchestrator. Two facts about it shape everything below:

1. It is declarative and it prunes. A controller loop reads a directory of service definitions from git and converges the cluster to match. If you delete a service block and push, the container is removed. A git push is a deploy.
2. Ingress, TLS, and routing are built in. Each node runs a reverse proxy, and certificates are issued per domain over HTTP-01.

The new personal cluster would be two nodes: one box as the controller, and an older second box joining later as an agent. The business cluster stayed exactly where it was.

The mechanic that matters: reconcile prunes

Because the controller continuously converges the cluster to git, every edit is live the moment it lands on the branch the controller watches. That is convenient and it is dangerous. The dangerous case here was moving a node that already ran personal containers from the old cluster into the new one. If I just deleted those services from the old cluster's git, the controller would happily remove the containers, and depending on how volumes were declared, the data could go with them.

The trick is ordering. To hand a node from cluster A to cluster B without losing data:

1. Stop the agent on that node first. A controller cannot act on a node it cannot reach. The containers keep running, the volumes stay put.
2. Only then remove those services from cluster A's git. The controller drops them from desired state but physically cannot prune the now-unreachable node.
3. Remove the old containers locally, keeping the volumes.
4. Install the new controller, re-set the secrets, deploy. The new services reattach to the surviving volumes by name.

Downtime for the whole cutover was about half an hour, most of it TLS reissue.

Secrets do not travel as files

The per-service secret store is encrypted at rest with a per-node key. Copying the encrypted file to the new controller would have produced garbage. The path that worked was to export the plaintext on the old controller, stage it, and re-import on the new one:

# on the old controller, per service
orca secrets get MY_KEY   # prints the decrypted value

# stage the values, then on the new controller
orca secrets import -f staged.env

The values never touched a terminal I did not control, and the encrypted blobs never left their node.

Moving object storage without a server-side copy

The buckets lived in one object storage project and needed to move to a separate one. There is no server-side cross-project copy on this provider, so the move was a streamed rclone copy, run on a machine in the same data center as the storage so the bytes never left the region:

rclone copy business:media personal:media --transfers 16 -P
rclone check business:media personal:media --one-way

rclone check reporting zero differences over several thousand objects was the green light. From there the app config pointed at the new bucket, and the old one was retired at the end.

The footgun: a mount that silently vanished

One service, a Navidrome music server, kept losing its library on every deploy. A roughly 26 GB corpus would reappear as empty. The cause was a single misplaced line in the TOML config:

[[service]]
name = "navidrome"
# ...
[service.volume]
path = "/data"
mounts = ["/host/music:/music"]   # WRONG: this is service.volume.mounts

In TOML, a key that appears after a [section] header belongs to that section. So mounts here parsed as service.volume.mounts, a field the orchestrator does not read. It never saw the bind, and the image's own VOLUME declaration created a fresh anonymous volume every deploy. The fix was to move mounts above the first sub-table so it belongs to the top-level service:

[[service]]
name = "navidrome"
mounts = ["/host/music:/music"]   # top-level, seen by the orchestrator
# ...
[service.volume]
path = "/data"

The lesson is boring and worth internalizing: in TOML, section headers are sticky. A key is owned by whatever [table] most recently opened above it.

What made this easy

Most of the personal subdomains already pointed at the node that was becoming the new controller, because those services already ran there. So the big cutover needed almost no DNS change. The proxy came up, HTTP-01 reissued certificates for each domain, and the sites were back. The only real DNS work was two records for services that migrated off the business box later.

Verification

The checks I actually ran after deploy, in order:

• Both databases present and listing their expected tables.
• The music bind resolving to the host path, not a fresh volume.
• Each domain returning a valid certificate and a 200.
• A trivial push producing an automatic redeploy through the webhook.
• The first scheduled backup landing in the new bucket.

That last one matters. A manual backup command only proved the local leg. Only the scheduled run proved the object storage upload path end to end.

Next

The cutover was the calm part. Right after it, two demo apps that share a domain started returning 404s at random, and webhook redeploys began failing with 503s. That sent me into the orchestrator's own Rust source, which is part two.