Back up Rancher before you upgrade: three ways the backup step failed in my 2.15.2 lab

Victor Estival — Thu, 05 Nov 2026 05:48:43 +0000

Every Rancher upgrade guide starts with "take a backup first". I ran that step in a lab in late October on Rancher 2.15.2 (biased, I work at SUSE) and it failed three different ways before it worked. Writing them down because none of them fails loudly.

The ResourceSet name. The Backup example I copied uses resourceSetName: rancher-resource-set. On my install that returned: resourcesets.resources.cattle.io "rancher-resource-set" not found. The operator ships two, rancher-resource-set-basic and rancher-resource-set-full. Run kubectl get resourcesets before you write the CR.
No storage, no backup. With no S3 details and no default storage location, the Backup sits in Retrying with "needs to specify S3 details, or configure storage location at the operator level". It doesn't go to Failed, so a quick look at the list can miss it.
The PVC needs a StorageClass. Installing rancher-backup with persistence.enabled=true alone left the PVC Pending on a cluster with no default StorageClass. Setting persistence.storageClass explicitly (local-path in my case) fixed it. With the right name and a volume, the backup reached Completed in under a minute.

Two more that are easy to skip:

If you encrypt backups, the operator doesn't keep your EncryptionConfiguration. You need that same file to restore, so store it somewhere that isn't the cluster.
Each Rancher chart declares a kubeVersion ceiling. helm show chart rancher-stable/rancher --version 2.13.3 | grep kubeVersion prints < 1.35.0-0. If the local cluster's Kubernetes moved ahead of Rancher, check this before the Helm upgrade.

What I didn't test: a restore, and a real minor-version jump (the lab already runs the latest release). If you've restored a Rancher backup for real, what surprised you?
First posted on r/rancher.

Scaling prod by hand during an incident, when Fleet manages it: three ways it goes, measured

Victor Estival — Wed, 14 Oct 2026 11:54:30 +0000

Every GitOps setup meets the same moment: something is on fire, someone runs kubectl scale on prod, and the fix works. What happens next depends on two settings most people never look at.

I ran all three cases in a lab on Rancher 2.15.1 with Fleet (biased, I work at SUSE): three EKS clusters, one app from Git.

1. Self-healing on (correctDrift)

I scaled a Deployment from 3 to 1. Fleet put it back to 3 in under 7 seconds. The GitRepo flickered NotReady for about 3 seconds, so nobody watching the UI would ever see the drift. Great for drift, useless during an incident: your fix is gone before you can check whether it worked.

2. Self-healing on, cluster paused

Continuous Delivery, Clusters, three-dot menu, Pause. I scaled to 5 and it held. But while the cluster was paused, the GitRepo read Ready, 2/2, no Modified state at all. The app looked healthy in the UI while one cluster ran something Git did not say. When I unpaused, the 5 went back to 3 within about 10 seconds. The fix lasted exactly as long as the pause.

3. Self-healing off

The manual scale stays, and Fleet reports it precisely: the Ready condition names the resource and the value Git wants. That part is good.

The surprise came with the next commit. I promoted a new chart version to prod and the upgrade failed. The manual scale had taken field ownership of spec.replicas (field manager "Mozilla", because it came from the browser, kubectl from a terminal), and server-side apply refused to hand it back. The UI showed WaitApplied. The actual conflict text was only in the failed Helm release on the cluster. Scaling back to the value in Git unblocked it, and the promotion landed about 40 seconds later.

What I'd take from it

Pause is a working break-glass. Use it, and keep in mind that while a cluster is paused, Continuous Delivery tells you nothing about it.
Put the fix in Git before you unpause, or you lose it.
With self-healing off, an emergency change that never makes it back to Git blocks your next deploy. If a promotion sits in WaitApplied after an incident, look for a field someone changed by hand.