What I Learned Studying EKS Cluster Upgrades (Beyond Just "Click Upgrade")

DevanshuRastogi — Sat, 29 Aug 2026 09:41:52 +0000

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades. My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too.

But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too.

Learning #1: There's No "Undo" Button

This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it.

So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after.

Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks

Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about, which might still be using an older API version internally.

There are tools built exactly for catching this before it becomes a problem:

pluto detect-helm -owide
pluto detect-files -d ./manifests

kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools exist for which job.

Learning #3: Addons Have Their Own Version Compatibility Rules

This one genuinely surprised me. Things like CoreDNS, kube-proxy, the VPC CNI plugin, and the EBS/EFS CSI drivers each have their own version, and that version needs to be compatible with whatever Kubernetes version you're running — it's not automatic.

You can actually check this directly:

aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.30

From what I read, a common mistake (and one that apparently causes confusing issues like pods failing to get IPs, or DNS acting weird) is upgrading the control plane and assuming the addons will "just work" because they were fine before. Apparently they need to be checked and upgraded as part of the same process, not treated as a separate afterthought.

Learning #4: Webhooks and Operators Can Break Things in Confusing Ways

This was the most interesting (and honestly a little intimidating) thing I came across. Tools like cert-manager, service meshes, or policy engines often register something called a ValidatingWebhookConfiguration or MutatingWebhookConfiguration. These sit directly in the path of every API request.

If one of these tools isn't fully compatible with the new Kubernetes version, the symptoms apparently don't look obviously related — you might just see random kubectl apply commands failing across the cluster, and it's not immediately obvious why.

The tip I picked up from reading around: before upgrading, it's worth just listing these out and knowing what owns each one:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Then checking each tool's own documentation for explicit support of the version you're upgrading to.

Learning #5: Node Upgrades Have Their Own Gotchas Too

I learned about PodDisruptionBudgets (PDBs) in this context, which was new to me. Apparently, if a PDB is configured too strictly (like requiring 100% of pods to stay available on something that only has one replica), it can cause node drains to hang or fail during an upgrade, because Kubernetes literally can't evict anything without violating the rule.

A pattern that seems to be recommended over "in-place" node upgrades is creating a new node group on the new version alongside the old one, gradually moving workloads over, and then removing the old node group — instead of upgrading nodes in place and hoping the timing works out.

The Bigger Lesson

The thing that stuck with me most after all this reading wasn't really a technical detail — it was more of a mindset shift. A lot of the "scary" upgrade stories I came across seem to happen when teams upgrade rarely (like once a year, only when AWS forces them to), which means a huge number of these small issues pile up at once.

It seems like the safer approach — at least from what I've read — is treating upgrades as a routine, smaller, more frequent thing rather than a big scary annual event. Smaller version jumps apparently mean fewer surprises each time.

My Own Study Checklist (for next time I practice this hands-on)

[ ] Run pluto or kubent against the target version before upgrading
[ ] Check addon compatibility (VPC CNI, CoreDNS, kube-proxy, CSI drivers) for the target version
[ ] List all webhook configurations and check if their owning tools support the new version
[ ] Check PodDisruptionBudgets for anything that could block node draining
[ ] Consider blue/green node groups instead of in-place upgrades
[ ] Remember: no control plane rollback — plan accordingly

I'm planning to actually simulate an upgrade hands-on soon using a local KIND cluster, since reading about it only gets you so far. If anyone experienced has caught mistakes in how I've understood any of this, I'd genuinely love the correction — still very much learning here.

VERSION CONTROL SYSTEM IN NUTSHELL

DevanshuRastogi — Sat, 27 Jan 2024 07:50:56 +0000

Introduction

Hello, fellow tech enthusiasts! Today, I want to delve into a topic that is integral to the world of software development - Version Control Systems (VCS), also known as Software Configuration Management or Source Control Management (SCM).

🚀 The Need for Version Control Systems

Imagine you're a developer working on a client project that contains 100 files. The client requests changes, and you dutifully make them. But then, the client wants to see the project as it was on Day 1. Without a VCS, you'd be helpless, as you didn't save or manage the code as it was on Day 1. This is why we shouldn't overwrite our code. In large organizations, where many developers are working on the same project, maintaining different versions of the project or task manually can be challenging. This is where VCS comes in.

💡 Who Uses Version Control Systems and Why?

It's not just developers who use VCS. Testers use it to manage their test scripts, architects to manage their design patterns in document forms, and project managers to manage their Excel sheets, among others.

🤝 How Version Control Systems Work

The working directory or workspace is where developers keep their files. It has nothing to do with version control. The repository is where we keep files along with their metadata. Here, version control is applicable. The client program, such as Git, is responsible for transferring files from the working directory to the repository and vice versa. The process of sending files from the working directory to the repository is called a commit, and each commit has a unique id, the commit id. The process of transferring files from the repository to the working directory is called checkout, sometimes referred to as pull or clone.

📖 The Benefits of Version Control Systems

VCS offers numerous benefits. We can maintain different versions of a file and choose any version as per our requirement. We can maintain metadata like the commit message, who made a commit, when a commit was made, and what changes have been made. It allows us to share code with different developers and collaborate, enabling parallel development. It also provides access control, determining who can read data and who can modify code.

🌟 Types of Version Control Systems

There are three types of VCS:

Local Version Control System

In a Local Version Control System, all the files are kept in different drives or locations. This makes it difficult to remember the exact location of the file, and if the file is deleted, the whole progress will be lost. This system is simple and easy to use, but it lacks the collaboration feature, which is crucial in a team environment.

Centralized Version Control System

In a Centralized Version Control System, there is a central repository where all files and their versions are saved. This allows different developers to contribute using a single repository. The process of taking code from the central repository to the local machine is called 'checkout', and the process of sending the code from the local machine to the central repository is called 'push'. However, this system has a single point of failure. If the central repository fails, the entire project history might be lost. Also, network connectivity should be available every time for operations.

Distributed Version Control System

In a Distributed Version Control System, every developer has their own local repository. They can commit or update the code in their local repository, and all users would be connected to a centralized remote repository in which they will push or pull the code. This means the version of the files or folders would be present in both the developers' repository and the central remote repository.

Conclusion

As I reflect on the importance of Version Control Systems, I'm filled with gratitude for the ease and efficiency they bring to our work as developers. They're the catalysts for effective collaboration and version management, reminding us that we're never alone on our coding journeys. So, until the next blog post, let's continue to learn, connect, and grow together. 🚀💫

DEV Community: DevanshuRastogi