Scaling prod by hand during an incident, when Fleet manages it: three ways it goes, measured

Every GitOps setup meets the same moment: something is on fire, someone runs kubectl scale on prod, and the fix works. What happens next depends on two settings most people never look at.

I ran all three cases in a lab on Rancher 2.15.1 with Fleet (biased, I work at SUSE): three EKS clusters, one app from Git.

1. Self-healing on (correctDrift)

I scaled a Deployment from 3 to 1. Fleet put it back to 3 in under 7 seconds. The GitRepo flickered NotReady for about 3 seconds, so nobody watching the UI would ever see the drift. Great for drift, useless during an incident: your fix is gone before you can check whether it worked.

2. Self-healing on, cluster paused

Continuous Delivery, Clusters, three-dot menu, Pause. I scaled to 5 and it held. But while the cluster was paused, the GitRepo read Ready, 2/2, no Modified state at all. The app looked healthy in the UI while one cluster ran something Git did not say. When I unpaused, the 5 went back to 3 within about 10 seconds. The fix lasted exactly as long as the pause.

3. Self-healing off

The manual scale stays, and Fleet reports it precisely: the Ready condition names the resource and the value Git wants. That part is good.

The surprise came with the next commit. I promoted a new chart version to prod and the upgrade failed. The manual scale had taken field ownership of spec.replicas (field manager "Mozilla", because it came from the browser, kubectl from a terminal), and server-side apply refused to hand it back. The UI showed WaitApplied. The actual conflict text was only in the failed Helm release on the cluster. Scaling back to the value in Git unblocked it, and the promotion landed about 40 seconds later.

What I'd take from it

• Pause is a working break-glass. Use it, and keep in mind that while a cluster is paused, Continuous Delivery tells you nothing about it.
• Put the fix in Git before you unpause, or you lose it.
• With self-healing off, an emergency change that never makes it back to Git blocks your next deploy. If a promotion sits in WaitApplied after an incident, look for a field someone changed by hand.

This is a lab finding, not official guidance. First posted on r/rancher: https://www.reddit.com/r/rancher/comments/1x0no93/scaling_prod_by_hand_during_an_incident_when/