DEV Community

Ashish Nair
Ashish Nair

Posted on

OpenShift Upgrade Deep Dive: 4.19 4.20 (Step-by-Step + What Happens Under the Hood)

#devops #infrastructure #kubernetes #tutorial

Upgrading Red Hat OpenShift isn’t just about running oc adm upgrade.
Behind the scenes, a carefully orchestrated process ensures your cluster stays available while everything gets updated.

In this guide, I’ll walk through:

  • Real Upgrade path 4.19 to 4.20
  • Handling AdminAckRequired blockers
  • Setting upgrade channels properly
  • Understanding how OpenShift upgrades nodes (masters + workers)
  • Why Pod Disruption Budgets (PDBs) can make or break your upgrade

Pre-Upgrade:

  • Check that none of the operators are "Degraded"
  • Check PDB's.
  • Check Failing Pods.
  • Run a must gather (oc adm must-gather).

1. Check current status

#oc adm upgrade
Enter fullscreen mode Exit fullscreen mode

The output will look something like:

Cluster version is 4.19.27

Upgradeable=False

Reason: AdminAckRequired
Message: The admissionregistration.k8s.io/v1beta1 group version is deprecated in 4.19 and will be removed in 4.20.Any clients using the v1beta1 version of these resources must be updated to use the corresponding v1 version instead. See https://access.redhat.com/articles/7130599 for more information.
Enter fullscreen mode Exit fullscreen mode

This essentially means our upgrade, usually Redhat will compliment the message with a knowledgebase as you can see in the above output.That knowledgebase will actually give you pointers on how to unblock.

2. Acknowledgement and Approval

The knowledgebase actually asked use to patch a configmap named admin-acks.

oc -n openshift-config patch cm admin-acks \
  --patch '{"data":{"ack-4.19-admissionregistration-v1beta1-api-removals-in-4.20":"true"}}' \
  --type=merge
Enter fullscreen mode Exit fullscreen mode

3. Setting the upgrade channel

At the time when I ran this upgrade updates were only available in the fast channel but In production consider using stable or EUS channels.

oc adm upgrade channel fast-4.20
Enter fullscreen mode Exit fullscreen mode

Verify by running the "oc adm upgrade" command again. Output should show something on these lines:

Channel: fast-4.20

VERSION     IMAGE
  4.20.18     quay.io/openshift-release-dev/ocp-release@sha256:2dab927fd20984e247301b2483083b71f942a1f550f5d8a1db42897edc042e39
Enter fullscreen mode Exit fullscreen mode

If I could explain the relevance of channels in one word:

Fast channels - Latest features but mostly used for Dev/Test environments
Stable channels - Production grade upgrades
EUS channels - LTS upgrades

4. Trigger the upgrade

oc adm upgrade --to=4.20.18
Enter fullscreen mode Exit fullscreen mode

Now, this is where most articles/guides stop. Because the next steps are taken care by openshift and most Engineers leave their desk for coffee breaks here. If you ask me, this part is equally important to understand, I will briefly walk though what openshift does "Under the Hood".

4.1 Upgrades the control planes first.

Suppose you have 3 masters, It will upgrade one master at a time, so that the cluster quorum is maintained. Each master goes through the below process:

  • Node gets cordoned.
  • Node gets drained.
  • Update is applied.
  • Node is rebooted.
  • API restored.
  • Uncordoned.
  • Moves on to the next Master.

4.2. Next it moves on the Worker nodes.

The worker nodes also go through a similar process:

  • Cordon
  • Drain
  • Update
  • Reboot
  • Uncordoned

Another topic that's less spoken about when an upgrade gets stuck is PDB's. If your PDB's are too strict your upgrade gets stuck and if it's too generous, the application is impacted. Be very careful while budgeting PDB's.

And lastly, Skip the coffee breaks and monitor the upgrade:

oc get clusterversion -w
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.19.27   True        True          11s     Working towards 4.20.18: 10 of 959 done (1% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18
version   4.19.27   True        True          11s     Working towards 4.20.18: 1 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 4 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 6 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 8 of 959 done (0% complete)
version   4.19.27   True        True          11s     Working towards 4.20.18: 10 of 959 done (1% complete)
Enter fullscreen mode Exit fullscreen mode

5. Verify completion

oc get clusterversion 
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.20.18   True        False         10m     Cluster version is 4.20.18
Enter fullscreen mode Exit fullscreen mode

My verdict on the upgrade process is, the upgrade is straight forward only when:

  • you handle any deprecations properly
  • Set correct channels.
  • Don't skip prechecks.

If this helped you, feel free to share or drop your upgrade experiences too!

Top comments (0)