DEV Community

Cover image for Everything Was Working. AWS Wanted $1,665/Month More.
AWS Community Builders profile image Mhamad El Itawi
Mhamad El Itawi for AWS Community Builders

Posted on

Everything Was Working. AWS Wanted $1,665/Month More.

#aws #mysql #kubernetes #devops

Nothing was down.

No alerts were firing. No customers were complaining.

CPU looked normal. Memory looked normal. The applications were serving traffic.

AWS just started charging more.

The first cost anomaly pointed to EKS extended support. A few days later, another one appeared for RDS MySQL extended support.

At first, this looked like a straightforward maintenance task: upgrade Kubernetes, upgrade MySQL, remove the extra charges.

It turned into something much more interesting.

The investigation uncovered years of infrastructure drift, Kubernetes components several versions behind the control plane, node groups that could not safely roll, disruption budgets that could deadlock maintenance, an application permanently scaled to maximum capacity because of JVM memory behavior, and several RDS Blue/Green blockers that were invisible until I actually tried the operation.

The systems had been working.

They just weren't as healthy as they appeared.

The alert came from the bill

The EKS anomaly was especially easy to understand once I looked at the usage type:

EU-AmazonEKS-Hours:extendedSupport
Enter fullscreen mode Exit fullscreen mode

That string tells you something important. This wasn't caused by more traffic, more pods, more nodes, or some workload suddenly consuming resources.

It was a calendar event. The Kubernetes version had reached the end of AWS standard support. The cluster hadn't changed, but the date had.

The EKS control-plane price went from roughly:

$0.10/hour → $0.60/hour
Enter fullscreen mode Exit fullscreen mode

A 6x increase.

The anomaly start date matched the end-of-standard-support date of the cluster version almost exactly.

The RDS anomaly had the same pattern:

EU-ExtendedSupport:Yr1-Yr2:MySQL8.0
Enter fullscreen mode Exit fullscreen mode

Again, nothing had suddenly become more expensive because of workload growth. A version simply became old enough for AWS to start charging for continued support.

AWS Cost Anomaly Detection did its job correctly. It flagged the charge.

Understanding why that charge appeared is still an engineering job.

The Kubernetes version wasn't the real problem

The EKS control plane was running Kubernetes 1.33.

That was easy enough to fix.

The more concerning part appeared when I started checking the rest of the cluster.

The update history showed a pattern like this:

2024: 1.27 → 1.28 → 1.29
2025: 1.30 → 1.31 → 1.32
2026: 1.33
Enter fullscreen mode Exit fullscreen mode

The control plane had been upgraded regularly.

The cluster add-ons had not.

Some of them still looked like they belonged to the original 2023 cluster.

A simplified view looked like this:

Component Running Expected around Kubernetes 1.33
kube-proxy 1.26 1.33
vpc-cni 1.12 1.22
CoreDNS 1.9 1.12
cluster-autoscaler 1.26 1.33

kube-proxy was seven Kubernetes minor versions behind the API server.

The cluster had been running like that for a long time.

And apparently running well enough that nobody had a reason to investigate it.

That is probably the most useful lesson from the whole exercise:

Working and supported are two very different states.

Why EKS didn't complain

I checked:

aws eks list-addons --cluster-name <CLUSTER>
Enter fullscreen mode Exit fullscreen mode

It returned nothing.

The add-ons weren't EKS-managed add-ons.

They were ordinary Deployments and DaemonSets living in kube-system.

That means EKS wasn't responsible for upgrading them. It also wasn't going to warn about the version skew.

This is an easy operational trap.

The control-plane upgrade is visible and simple:

aws eks update-cluster-version ...
Enter fullscreen mode Exit fullscreen mode

The add-ons are a different lifecycle.

If nobody explicitly owns that lifecycle, you can upgrade the control plane several times while networking, DNS, autoscaling, and proxy components stay exactly where they were.

Before upgrading anything, I checked what would break

The tempting approach was to start with the versions.

Instead, I spent time checking whether the cluster could actually survive a node replacement.

That turned out to be more valuable than the upgrade itself.

Three PDBs would have blocked node drains

Three Vertical Pod Autoscaler components had PodDisruptionBudgets like this:

minAvailable: 1
replicas: 1
allowed disruptions: 0
Enter fullscreen mode Exit fullscreen mode

That combination effectively says:

This pod may never be voluntarily evicted.

Which is a problem when an EKS node-group upgrade needs to drain the node.

My first instinct could have been to scale the components to two replicas.

That would have been the wrong fix.

The VPA version in use did not have the leader-election behavior I wanted to rely on. Two updater instances could potentially act independently, and two recommenders could create conflicting behavior.

So I checked the admission webhook:

kubectl get mutatingwebhookconfiguration \
  -o custom-columns=NAME:.metadata.name,POLICY:.webhooks[*].failurePolicy
Enter fullscreen mode Exit fullscreen mode

It used:

failurePolicy: Ignore
Enter fullscreen mode Exit fullscreen mode

A short VPA interruption during the node drain was therefore acceptable.

The safer fix was to relax the PDB instead:

kubectl patch pdb <name> -n kube-system --type=json \
  -p '[
    {"op":"remove","path":"/spec/minAvailable"},
    {"op":"add","path":"/spec/maxUnavailable","value":1}
  ]'
Enter fullscreen mode Exit fullscreen mode

Now Kubernetes could temporarily evict one pod during maintenance.

The node group had no room to move

The managed node group looked roughly like this:

minSize:     3
desiredSize: 3
maxSize:     3
Enter fullscreen mode Exit fullscreen mode

The cluster was pinned at exactly three nodes.

That sounds controlled, but it removes an important safety margin.

Two separate systems may need capacity above the current node count:

  1. Cluster Autoscaler, when pods become unschedulable.
  2. EKS, when it creates replacement nodes during a rolling node-group upgrade.

I raised the ceiling:

maxSize: 6
Enter fullscreen mode Exit fullscreen mode

and changed the update configuration to:

maxUnavailable: 1
Enter fullscreen mode Exit fullscreen mode

instead of a percentage.

A small but useful operational point here:

maxSize does not cost money. Running nodes cost money.

minSize affects your guaranteed footprint.

maxSize is just a ceiling.

Setting the ceiling equal to the current node count does not save anything. It just removes your emergency headroom.

This change ended up helping twice later that same day.

The application had no PDB

The main API deployment had several replicas and an HPA, but no PodDisruptionBudget.

So during a drain, Kubernetes had no application-level guarantee preventing too many API pods from disappearing together.

I initially wrote:

minAvailable: 5
Enter fullscreen mode Exit fullscreen mode

There were seven pods at the time, so that looked reasonable.

Then I checked the HPA:

minReplicas: 3
maxReplicas: 7
Enter fullscreen mode Exit fullscreen mode

My PDB was wrong.

If the HPA scaled down to three pods overnight, a minAvailable: 5 PDB would become impossible to satisfy.

Allowed disruptions would become zero.

In other words, I was about to introduce the same drain deadlock I had just fixed elsewhere.

I changed it to:

maxUnavailable: 1
Enter fullscreen mode Exit fullscreen mode

That works throughout the HPA's scaling range.

I've become fairly cautious about fixed minAvailable values on deployments controlled by an HPA for exactly this reason.

Then the Helm problem appeared

I created the PDB manually with kubectl.

The application itself was deployed through Helm.

That means the next time the Helm chart introduced the same PDB, the deployment pipeline would fail because Helm did not own the existing resource.

The error would look something like:

invalid ownership metadata;
missing key "app.kubernetes.io/managed-by": must be set to "Helm"
Enter fullscreen mode Exit fullscreen mode

So I explicitly transferred ownership:

kubectl label pdb <name> -n <namespace> \
  "app.kubernetes.io/managed-by=Helm" --overwrite

kubectl annotate pdb <name> -n <namespace> \
  "meta.helm.sh/release-name=<release>" \
  "meta.helm.sh/release-namespace=<namespace>" \
  --overwrite
Enter fullscreen mode Exit fullscreen mode

This was a good reminder that infrastructure changes have two states:

  1. what exists in the cluster right now
  2. what the system of record thinks should exist

Fixing only the first one is temporary.

Find the owner before you patch the resource

That became another theme during the work.

Some resources were managed by Terraform.

Some were managed by Helm releases declared inside Terraform.

Some were deployed by CI.

Others had simply been created manually years ago.

A live kubectl patch can make production healthier immediately while also creating future drift.

For example, Terraform still pinned the node group to an old Kubernetes version even though the cluster was several versions ahead.

A future terraform apply could therefore try to reconcile production back toward stale configuration.

The fix wasn't just updating production.

The source of truth had to be updated too.

Otherwise the repair had an expiry date.

While preparing the upgrade, I found another cost problem

While checking whether the application would behave properly during pod rescheduling, I noticed that all of the API pods had recent Out Of Memory (OOM) kills.

This wasn't caused by the upgrade work. The OOMs had started many hours earlier.

But it mattered because a node rollout creates scheduling churn, and unstable memory behavior is exactly the kind of thing I don't want to discover halfway through draining production nodes.

The root cause was an older JVM build running inside a container with a memory limit.

That JVM predated reliable cgroup v2 awareness.

So instead of correctly seeing the container's memory budget, it saw something closer to the host's available memory and sized its heap accordingly.

The container limit was around 8 GiB.

The JVM ended up believing it could use almost all of that for heap alone.

That left very little room for:

  • direct buffers
  • metaspace
  • JIT memory
  • thread stacks
  • networking libraries
  • other native allocations

Eventually the kernel killed the container.

The fix was intentionally boring:

-Xmx5g
Enter fullscreen mode Exit fullscreen mode

Explicit heap ceiling. Plenty of room left for native memory.

After the change, memory usage dropped dramatically.

But there was a second effect I hadn't initially connected to the OOM issue.

The HPA used memory as one of its scaling signals.

Because every pod stayed above the configured memory target, the deployment had been pinned at its maximum replica count.

Once heap usage became sane, the HPA gradually scaled the API from seven pods down to three.

So a missing JVM memory limit wasn't only causing crashes.

It had also been keeping the application at maximum capacity around the clock.

That is one of those bugs where infrastructure, runtime behavior, and cost management all meet in the same place.

Adopting the EKS add-ons

Once the cluster was safe to drain, I started fixing the add-on drift.

I deliberately went one component at a time.

Not because these upgrades are particularly exotic, but because changing networking, DNS, proxying, and autoscaling simultaneously is a bad debugging strategy.

The order was:

  1. kube-proxy
  2. CoreDNS
  3. VPC CNI
  4. Cluster Autoscaler and the remaining components

Before using OVERWRITE, inspect what you're overwriting

Converting the self-managed components to EKS-managed add-ons required:

--resolve-conflicts OVERWRITE
Enter fullscreen mode Exit fullscreen mode

That flag deserves some respect.

AWS may replace local configuration with the managed add-on defaults.

So before doing it I checked the existing configuration.

For example:

kubectl get daemonset aws-node -n kube-system \
  -o jsonpath="{range .spec.template.spec.containers[0].env[*]}{.name}={.value}{'\n'}{end}"
Enter fullscreen mode Exit fullscreen mode

CoreDNS:

kubectl get configmap coredns -n kube-system \
  -o jsonpath="{.data.Corefile}"
Enter fullscreen mode Exit fullscreen mode

And kube-proxy:

kubectl get configmap kube-proxy-config -n kube-system \
  -o jsonpath="{.data.config}"
Enter fullscreen mode Exit fullscreen mode

The important finding was that they were effectively stock configurations.

That changed the risk assessment considerably.

Overwriting a heavily customized CoreDNS configuration is one thing.

Replacing a default-ish configuration with AWS's supported default is another.

The CNI service account also wasn't using IRSA, so I intentionally avoided providing a service-account role ARN during adoption. Otherwise the migration itself would have changed its AWS identity.

Then the add-ons could be adopted:

aws eks create-addon \
  --cluster-name <CLUSTER> \
  --addon-name kube-proxy \
  --addon-version <VERSION> \
  --resolve-conflicts OVERWRITE \
  --region <REGION>
Enter fullscreen mode Exit fullscreen mode

The only Kubernetes blip was DNS

Most of the add-on upgrades were uneventful.

CoreDNS caused a very small interruption.

During the pod replacement, there was roughly a two-second DNS gap. Two background Redis endpoint resolutions failed and recovered.

The slightly funny part was that the cause was the old CoreDNS configuration.

It had no lameduck period.

When the old pod began terminating, it could stop answering while still briefly appearing behind the Service.

The new managed configuration included both lameduck and the ready plugin.

So the upgrade caused the one small issue that the upgrade itself also fixed.

Future CoreDNS rollouts should behave better.

Don't blindly trust EKS upgrade insights

After fixing kube-proxy, AWS still reported an upgrade insight saying kube-proxy had unacceptable version skew.

That looked worrying until I checked the timestamps.

The insight had been calculated many hours before the add-on repair.

It was stale.

The live state was already correct:

kubectl get daemonset kube-proxy ...
kubectl version ...
Enter fullscreen mode Exit fullscreen mode

The versions matched.

AWS upgrade insights are useful, but they aren't necessarily live.

If an insight says ERROR, check when it was last refreshed before treating it as current truth.

Control plane upgrades aren't where most of the risk lives

There is an important distinction in EKS upgrades.

Updating the control plane:

aws eks update-cluster-version ...
Enter fullscreen mode Exit fullscreen mode

does not drain your worker nodes.

Your existing pods keep running.

The larger operational event is the managed node-group update.

That is when EKS has to:

  • create replacement nodes
  • cordon old ones
  • evict workloads
  • respect PDBs
  • reschedule pods
  • terminate old instances

The control-plane upgrade completed with zero application pod restarts.

Then came the node-group roll.

And that's where the earlier preparation paid off.

The maxSize change paid for itself twice

During an unrelated application deployment in the middle of the work, a few pods became Pending because of insufficient CPU.

Cluster Autoscaler added a fourth node.

The pods scheduled successfully.

Had maxSize still been three, the rollout would have stalled.

Later, during the managed node-group upgrade, EKS temporarily created two replacement nodes before draining the old ones.

For a while, the cluster had five nodes.

Again, impossible under the old ceiling.

What had looked like a theoretical safety change a few hours earlier solved two real capacity problems on the same day.

The node rollout completed without API pod restarts.

The PDB, topology placement, and extra node-group headroom did exactly what they were supposed to do.

Then came MySQL

The RDS side was a different kind of problem.

The database was running MySQL 8.0 and had entered extended support.

It was also Single-AZ.

For a major-version in-place upgrade, that meant a potentially long hard outage.

So I chose RDS Blue/Green Deployments.

The idea is straightforward:

  • create a green environment from production
  • keep it replicating
  • upgrade green
  • test it
  • switch over when satisfied

The final switchover should take seconds rather than tens of minutes.

More importantly, the risky work happens before you commit.

If green doesn't behave properly, you can walk away.

That, to me, is the real value of Blue/Green.

Upstream MySQL documentation nearly sent me down the wrong path

The application users authenticated using mysql_native_password.

The newer MySQL major version changes the default behavior around that plugin.

At first glance, that looked like an outage waiting to happen.

If the plugin wasn't available after the upgrade, every existing application connection could fail.

Before planning an authentication migration, I checked the actual RDS parameter family for the target engine version instead of relying only on upstream MySQL defaults.

RDS reported the equivalent of:

mysql_native_password = ON
modifiable = false
Enter fullscreen mode Exit fullscreen mode

AWS deliberately keeps it enabled.

That materially changed the migration plan.

An entire authentication migration disappeared once I checked the provider behavior instead of relying only on upstream release notes.

Later, I verified it the better way: actually connecting to the green database with the existing credentials.

Documentation tells you what should work.

A real connection proves what does work.

Do we really need a primary key on every table?

Another concern was row-based replication.

Several tables did not have primary keys.

Most were tiny.

One was a multi-million-row Spring Batch metadata table.

Typical advice for this situation is simple:

Add primary keys before using replication.

That would have meant altering a large production table before the migration.

Instead, I looked at why the recommendation exists.

With row-based replication, the replica needs an efficient way to locate rows for UPDATE and DELETE.

For an insert-only table, that concern is much smaller.

This particular table contained job parameters. Rows were inserted when jobs started and effectively never updated. Deletions only happened during manual cleanup.

Rather than alter production preemptively, I decided to create the Blue/Green environment and measure replication lag.

If it couldn't keep up, green was disposable.

The lag looked roughly like:

1.0s
0.0s
0.0s
0.0s
0.0s
Enter fullscreen mode Exit fullscreen mode

There was no practical replication problem.

No ALTER TABLE was needed.

Rules are useful.

Understanding the reason behind the rule is more useful.

Blue/Green had three blockers waiting

This was probably the most AWS part of the whole exercise.

Nothing in the visible configuration suggested the migration couldn't proceed.

Then I ran the command.

Blocker 1: Secrets Manager

RDS returned:

SourceDatabaseNotSupportedFault:
Databases using Secrets Manager are not currently supported
for Blue Green Deployments
Enter fullscreen mode Exit fullscreen mode

The master password was RDS-managed through Secrets Manager.

Rotation was disabled.

I reverted the database back to a self-managed master password while keeping the same effective credential already in use.

No restart was needed.

Then Blue/Green could proceed.

Blocker 2: custom option group

The next attempt failed:

InvalidParameterCombination:
RDS Blue/Green Deployments only support default option groups
for major version upgrades.
Enter fullscreen mode Exit fullscreen mode

AWS suggested creating green on the existing major version first and upgrading green afterward.

That was preferable anyway.

The production option group contained no active options, but I still preferred the path that did not require changing production.

So the flow became:

  1. Create green on the current MySQL version.
  2. Upgrade green separately to the target version.

Blocker 3: the target option group didn't exist

Then another error appeared:

OptionGroupNotFoundFault:
Specified OptionGroupName: <TARGET_OPTION_GROUP> not found.
Enter fullscreen mode Exit fullscreen mode

There had never been an instance running the target engine version in that account.

The required option group wasn't there.

So I created one.

Three blockers.

None were discovered during the initial review.

All three appeared only when attempting the real operation.

That's why I now budget migration time for discovery, not just execution.

Cloud APIs often contain constraints you don't see until you reach them.

The parameter group was easier to miss than the hard errors

Hard errors are annoying, but at least they're visible.

A silent configuration change is worse.

If I had created the green database using the defaults, some custom MySQL settings would have disappeared.

One of them was slow query logging.

The migration could have "worked" while quietly changing production observability after switchover.

So before creating the final environment, I recreated the required settings in a parameter group for the target engine family.

Major-version upgrades aren't just about whether the database boots.

They are also configuration migrations.

Validation had three layers

I didn't want the final check to be:

MySQL says available, let's switch.

I used three levels of validation.

1. Database-level comparison

I connected to both blue and green and checked:

  • expected custom parameter values
  • table count
  • sampled row counts
  • authentication using the existing credentials

The sampled counts matched exactly.

Both databases had the same number of tables.

2. Boot the actual production application against the upgraded database

Next I launched the actual production application image and pointed it at green.

For Hibernate, I used:

ddl_auto=validate
Enter fullscreen mode Exit fullscreen mode

not:

ddl_auto=update
Enter fullscreen mode Exit fullscreen mode

That distinction matters.

I wanted the application to fail if the schema was incompatible.

I did not want the validation pod "fixing" the schema by issuing DDL against green and making the comparison meaningless.

A successful application boot gave me several useful signals at once:

  • JDBC connectivity worked
  • authentication worked
  • the driver handled the upgraded MySQL server
  • Hibernate detected the database successfully
  • the mapped schema validated
  • application queries parsed correctly

The test pod was deliberately kept outside the production Service selector so it could never receive live requests.

I also disabled the background consumers that could reasonably be disabled for the test.

3. Send the same real request to both environments

This was the strongest test.

I took a real authenticated API request and sent it to:

  • the current production application connected to the original database
  • the test instance connected to the upgraded green database

Then I diffed the responses.

They were byte-for-byte identical.

That's much more useful than saying:

The application started.

The goal of a database upgrade isn't just to make the application boot.

The goal is to keep the application behaving the same way.

The final switchover

When the checks were done, I started the Blue/Green switchover.

Timeline:

07:22:21  switchover started
07:23:27  switchover completed
07:23:37  API responding normally
Enter fullscreen mode Exit fullscreen mode

About 66 seconds.

The application pods did not restart.

There were no database errors in the application logs.

HikariCP re-established the database connections transparently.

The same API request used before the switchover returned the same data afterward.

I had expected some temporary latency increase because the upgraded environment had a colder InnoDB buffer pool.

The effect wasn't meaningful.

That prediction simply didn't materialize at this workload and database size.

Eleven minutes later, the client's team pushed a normal application release through the regular CI pipeline.

It built, deployed, and connected to the upgraded database without any special handling.

That was probably a better validation than my test pod.

The result

At the end of the work:

Component Before After
Kubernetes control plane 1.33 upgraded and back in standard support
Worker kubelets 1.33 aligned with the control plane
kube-proxy 1.26 current
CoreDNS old current
VPC CNI old current
Cluster Autoscaler 1.26 current
MySQL 8.0, extended support upgraded and back in standard support
API replicas 7 3
Extended-support charges active removed

The Kubernetes work included:

  • three major add-on refreshes
  • a control-plane upgrade
  • a complete managed-node-group replacement
  • an autoscaler upgrade spanning several Kubernetes versions
  • new application disruption protection
  • node-group capacity changes
  • a JVM memory fix

None of that restarted the API pods.

The only actual service interruption was the RDS Blue/Green switchover, at roughly one minute.

There was also a tiny DNS blip during the CoreDNS adoption, limited to a couple of background lookups.

The whole exercise took roughly six hours.

What I took away from it

The most interesting part wasn't the EKS command or the RDS command.

Those were easy.

The useful lessons were around everything surrounding them.

Working isn't the same as supported.

A cluster can run happily with components far outside their supported version skew. Lack of incidents is not proof that the platform is being maintained properly.

Control-plane-only Kubernetes upgrades are dangerous operationally because they feel complete.

The command succeeds, the cluster reports the new version, and it's easy to forget the components actually running on the worker nodes.

Check the support window of the version you're upgrading to.

There is little value in completing an emergency upgrade only to land on another version that reaches end of standard support a few months later.

PDBs need to match your scaling model.

A fixed minAvailable that looks safe at seven replicas can deadlock maintenance when the HPA scales the workload down to three.

Figure out who owns a Kubernetes object before changing it.

Terraform, Helm, CI and manual kubectl changes all behave differently. Production state and declared state need to agree.

Leave node groups some ceiling.

maxSize is not a cost optimization knob. Setting it too low removes the exact headroom you need during incidents, deployments, and upgrades.

Managed database services are not identical to upstream software.

RDS defaults saved me from doing an unnecessary authentication migration. Always inspect the provider behavior you're actually running.

Understand why operational rules exist.

The primary-key recommendation for row replication exists for a reason. Once you understand the reason, you can evaluate an unusual table based on its actual write pattern rather than following the rule mechanically.

Provider insights may be cached.

Always compare advisory tools against live state before making a production decision.

And finally:

Budget time for things you haven't discovered yet.

The RDS upgrade produced three separate blockers. None were visible during the initial inspection. None were particularly difficult to solve.

But they only became visible once the real commands were executed.

That pattern showed up repeatedly throughout the day.

The configuration that blocks your upgrade is often invisible until you actually try to upgrade.

Which brings the story back to where it started.

Nothing had failed.

Everything looked normal.

The bill was the only alarm.

🌐 For more engineering stories and practical lessons, follow me on LinkedIn or Instagram.

Top comments (0)