DEV Community

Florient DOGBE
Florient DOGBE

Posted on

Upgrading an OpenShift Cluster Without Waking the On-Call: Channels, MCPs and Controlled Pauses

#kubernetes #openshift #sre #devops

An OpenShift upgrade is two very different operations wearing the same trench coat. The first one is a control-plane operator rollout driven by the Cluster Version Operator (CVO): fast, mostly invisible, and reversible only in narrow cases. The second one is a fleet-wide node reboot campaign driven by the Machine Config Operator (MCO): slow, disruptive, and entirely at the mercy of your PodDisruptionBudgets.

On a 12-node lab cluster nobody notices. On a 300-node cluster with stateful workloads, a default maxUnavailable: 1 on the worker pool means a reboot campaign that runs for 20 hours straight — right across your 3 a.m. quiet window, with no natural checkpoint. That is how upgrades end up paging the on-call.

This article breaks down the machinery — ClusterVersion, update channels, MachineConfigPool — and shows how to turn a monolithic upgrade into a wave-based rollout with real maintenance windows and a usable abort path.

Context: the two clocks of an OpenShift upgrade

The CVO reconciles a single cluster-scoped object, ClusterVersion/version. You declare a target payload; the CVO walks the manifests of the release image in dependency order, updating every ClusterOperator. Control-plane nodes get their new RHCOS config from the MCO as part of that process, and because there are only three of them with maxUnavailable: 1 hardcoded, it is bounded and predictable.

Then the worker pools start. The MCO renders a new MachineConfig per pool, cordons a node, drains it, applies the OS update via rpm-ostree, reboots, uncordons, and moves to the next. The CVO reports Completed for the version long before the last worker has rebooted — oc get clusterversion will happily show 4.16.21 while 180 nodes are still pending. That gap is exactly where unplanned incidents live.

The lever you control is the MachineConfigPool. Everything below is about using it.

The CVO side: ClusterVersion and channels

Reading the graph before touching anything 

oc adm upgrade
oc get clusterversion version -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}'
oc get co
Enter fullscreen mode Exit fullscreen mode

Three things matter in that output:

  • Upgradeable=False blocks minor upgrades. The message tells you why: a pending admin ack for removed Kubernetes APIs, an operator in a bad state, or an OLM operator with no compatible version for the target.
  • Recommended vs conditional updates. oc adm upgrade only lists recommended targets. Use oc adm upgrade --include-not-recommended to see conditional ones along with the risk reason (a known bug affecting a specific platform or configuration). Read the risk before you accept it.
  • Degraded ClusterOperators. The CVO refuses to start if an operator is Degraded=True, unless you pass --allow-upgrade-with-warnings. Don't.

Channel semantics

Channels are not "quality tiers" in the sense people assume. A given 4.16.z release lands in candidate-4.16, then fast-4.16, then stable-4.16 after a soak period driven by telemetry. The payload is identical; only the recommendation timing differs. eus-4.16 exists only for even-numbered minors and carries the extended-support stream plus the edges needed for EUS-to-EUS hops.

A sane fleet policy: non-production on fast-4.y, production on stable-4.y, and EUS-pinned clusters on eus-4.y. Switching channel is declarative and non-destructive:

# Move to the next minor's channel to expose the upgrade edges
oc adm upgrade channel stable-4.16

# Target a specific patch, never "latest"
oc adm upgrade --to=4.16.21
Enter fullscreen mode Exit fullscreen mode

Always pin the exact version. --to-latest is a moving target between the moment you write the change request and the moment you execute it.

If you need to abort before any manifest has been applied:

oc adm upgrade --clear
Enter fullscreen mode Exit fullscreen mode

This only works while the update has not actually started rolling. Once ClusterOperators begin moving, the forward path is the only path.

The MCO side: MachineConfigPool as a rollout controller

maxUnavailable is your throughput dial

The worker pool defaults to maxUnavailable: 1. On a large cluster, compute your window budget first:

window_duration / (drain_time + reboot_time) * maxUnavailable = nodes per window
Enter fullscreen mode Exit fullscreen mode

With a 12-minute average per node (drain included) and a 4-hour window, maxUnavailable: 1 gives you 20 nodes. maxUnavailable: 10% on a 200-node pool gives 20 concurrent reboots — far too aggressive for most workloads. Something between 3 and 8 % is usually the sweet spot, but validate against your PDBs:

oc get pdb -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,MIN:.spec.minAvailable,MAX:.spec.maxUnavailable,ALLOWED:.status.disruptionsAllowed'
Enter fullscreen mode Exit fullscreen mode

Any PDB with disruptionsAllowed: 0 will stall a drain indefinitely. The MCO retries the drain for an hour, then marks the pool Degraded with NodeDrainFailed. Fix those before the window, not during.

Custom pools are your waves

A node carrying a custom role label leaves the worker pool and joins the matching custom pool, which inherits worker MachineConfigs through machineConfigSelector. That is the primitive for wave-based rollouts:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: wave-a
spec:
  # Pick up every rendered worker config, plus anything scoped to wave-a
  machineConfigSelector:
    matchExpressions:
      - key: machineconfiguration.openshift.io/role
        operator: In
        values:
          - worker
          - wave-a
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/wave-a: ""
  maxUnavailable: 2
  paused: true
Enter fullscreen mode Exit fullscreen mode 
# Canary wave: 5 nodes spread across failure domains
for n in worker-a-01 worker-b-01 worker-c-01 worker-a-02 worker-b-02; do
  oc label node "$n" node-role.kubernetes.io/wave-a=
done

oc get mcp wave-a -o jsonpath='{.status.machineCount}{"\n"}'
Enter fullscreen mode Exit fullscreen mode

Create wave-a, wave-b, wave-c (and leave the remainder in worker), all with paused: true. A node must never match two custom pools — the MCO will mark it degraded with node has been in multiple pools.

One caveat on labelling: moving a node between pools triggers a config change and therefore a reboot. Do the labelling work before the upgrade, in a quiet period, so that upgrade day only flips paused.

Orchestrating the rollout

Phase 0 — preflight, D-2 

# etcd backup on a control-plane node
oc debug node/master-0 -- chroot /host /usr/local/bin/cluster-backup.sh /home/core/assets/backup

# No firing alerts that would mask upgrade symptoms
oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- \
  curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="firing") | .labels.alertname' | sort -u

# Admin acks for removed APIs (key name is in the Upgradeable message)
oc -n openshift-config patch cm admin-acks --type=merge \
  -p '{"data":{"ack-4.15-kube-1.29-api-removals-in-4.16":"true"}}'
Enter fullscreen mode Exit fullscreen mode

Also confirm every OLM-managed operator has a channel compatible with the target minor; an operator stuck on an incompatible channel is a classic post-upgrade outage.

Phase 1 — pause the fleet, upgrade the control plane 

for p in worker wave-a wave-b wave-c; do
  oc patch mcp/$p --type=merge -p '{"spec":{"paused":true}}'
done

oc adm upgrade channel stable-4.16
oc adm upgrade --to=4.16.21

# Follow progress (oc 4.16+ ships a richer status view)
watch -n 30 'oc get clusterversion version; oc get co | grep -v "True        False         False"'
Enter fullscreen mode Exit fullscreen mode

With all worker pools paused, the blast radius is the control plane only: three sequential master reboots, operators rolled out, zero worker drain. This typically fits in a 90-minute window.

Phase 2 — canary wave 

oc patch mcp/wave-a --type=merge -p '{"spec":{"paused":false}}'

watch -n 20 'oc get mcp wave-a -o custom-columns=\
NAME:.metadata.name,UPDATED:.status.updatedMachineCount,\
TOTAL:.status.machineCount,DEGRADED:.status.degradedMachineCount'
Enter fullscreen mode Exit fullscreen mode

Per-node visibility when something stalls:

oc get nodes -l node-role.kubernetes.io/wave-a -o custom-columns=\
'NAME:.metadata.name,STATE:.metadata.annotations.machineconfiguration\.openshift\.io/state,\
CURRENT:.metadata.annotations.machineconfiguration\.openshift\.io/currentConfig,\
DESIRED:.metadata.annotations.machineconfiguration\.openshift\.io/desiredConfig'

# machine-config-daemon logs for a stuck node
oc -n openshift-machine-config-operator logs -l k8s-app=machine-config-daemon \
  --field-selector spec.nodeName=worker-a-01 --tail=200
Enter fullscreen mode Exit fullscreen mode

Then bake. Give the canary 24 to 48 hours of real traffic before opening the next wave. That bake time is the entire point of the exercise.

Phase 3 — remaining waves

Unpause one pool per maintenance window, and re-pause it at the end of the window if it has not finished:

# End of window: freeze whatever is left, mid-flight nodes finish cleanly
oc patch mcp/wave-b --type=merge -p '{"spec":{"paused":true}}'
Enter fullscreen mode Exit fullscreen mode

Pausing mid-rollout does not interrupt a node already draining; it only stops the MCO from picking up new ones. That is your natural checkpoint — and the reason the on-call phone stays quiet.

EUS-to-EUS: skipping a minor without double-draining

EUS channels let you go from 4.14 to 4.16 with a single worker reboot campaign instead of two. The mechanics are exactly the pause trick, applied across two consecutive control-plane upgrades:

# 1. Pause every worker-bearing pool
for p in worker wave-a wave-b wave-c; do
  oc patch mcp/$p --type=merge -p '{"spec":{"paused":true}}'
done

# 2. 4.14 -> 4.15 (control plane only)
oc adm upgrade channel eus-4.16
oc adm upgrade --to=4.15.44

# 3. 4.15 -> 4.16 (control plane only)
oc adm upgrade --to=4.16.21

# 4. Only now, unpause wave by wave
oc patch mcp/wave-a --type=merge -p '{"spec":{"paused":false}}'
Enter fullscreen mode Exit fullscreen mode

Two constraints are non-negotiable. First, the control plane can only ever be two minors ahead of the kubelet (the Kubernetes version-skew policy), so you cannot chain three minors. Second, a paused pool stops receiving the rotated kubelet client CA. The MachineConfigControllerPausedPoolKubeletCA alert fires when that becomes dangerous; if you ignore it long enough, nodes lose their ability to authenticate to the API server. Treat a paused pool as a resource with a TTL measured in days, not weeks, and track it explicitly.

Rollback: what is actually possible

Be honest with the change advisory board: a completed minor upgrade is not reversible. CRD schema migrations, etcd storage version bumps and operator-managed data changes have no downgrade path. What you do have:

  • Abort before start — oc adm upgrade --clear, valid only while no manifest has been applied.
  • Z-stream rollback — recent oc builds expose oc adm upgrade rollback, restricted to returning to the previously installed patch release within the same minor. Check oc adm upgrade --help on your client version before relying on it.
  • Partial rollback at pool level — this is the realistic one. Pools still paused have never left the old OS version. If the canary wave misbehaves, you keep 95 % of the fleet on the known-good RHCOS image while you investigate, and the cluster keeps running in a supported skew.
  • MachineConfig rollback — deleting a custom MachineConfig makes the MCO render the previous configuration and roll the pool back to it. This covers your own kernel args, chrony or kubelet tweaks, not the OS payload itself.

On 4.17+, nodeDisruptionPolicy removes reboots entirely for many file and unit changes, shrinking the rollback surface:

apiVersion: operator.openshift.io/v1
kind: MachineConfiguration
metadata:
  name: cluster
spec:
  nodeDisruptionPolicy:
    files:
      - path: /etc/chrony.conf
        actions:
          - type: Reload
            reload:
              serviceName: chronyd.service
Enter fullscreen mode Exit fullscreen mode

Security best practices

Never use --force. It bypasses release-image signature verification and precondition checks. If oc adm upgrade refuses a target, the answer is to fix the precondition, not to disable the guardrail. For disconnected clusters, verify the mirrored payload explicitly and declare mirrors through ImageDigestMirrorSet so signature validation still resolves.

Lock down who can write to ClusterVersion and MachineConfigPool. Both are cluster-scoped and both are production-destroying. patch on machineconfigpools is effectively "reboot the fleet" or "silently stop security patches" — audit it like a privileged credential and restrict it to a dedicated upgrade ServiceAccount used by your pipeline.

Treat paused: true as a security exception. A paused pool stops receiving RHCOS CVE fixes and certificate rotations. Add a monitoring rule that alerts on any pool paused for more than 72 hours, and require a ticket reference in the pool annotations.

Do not use spec.overrides in ClusterVersion to silence a stuck operator. It makes the CVO stop managing that component — including its security updates — and it sets Upgradeable=False for everything that follows.

Handle admin acks deliberately. ack-*-api-removals-* keys exist because workloads still calling removed APIs will break. Query the apirequestcounts resource (oc get apirequestcounts -o json) to identify the callers before acking — not after.

Protect the etcd backup. It contains every Secret in the cluster in plaintext unless etcd encryption is enabled. Encrypt at rest, restrict access, and set a retention policy.

Key takeaways

  • ClusterVersion drives the control plane; MachineConfigPool drives the nodes. They complete at very different times, and only the second one touches your workloads.
  • Pin an exact patch version, read conditional-update risks, and clear Upgradeable=False properly instead of forcing through it.
  • Pre-create custom pools (wave-a, wave-b, …) with tuned maxUnavailable, label nodes ahead of time, and upgrade by unpausing one pool per maintenance window.
  • Pausing a pool is a checkpoint, not a parking spot: kubelet CA rotation and CVE patching stop with it.
  • EUS-to-EUS halves your reboot campaigns, at the cost of a longer pause window you must actively monitor.
  • Rollback is a pool-level concept, not a cluster-level one. Design the waves so that "stop here" is always a valid, supported state.

Top comments (0)