DEV Community: Silpa Nallagopula

PostgreSQL 14 ends on Nov 12. The upgrade won't break your Java service. The next password reset will.

Silpa Nallagopula — Fri, 06 Nov 2026 16:02:20 +0000

PostgreSQL 14 gets its final release on November 12. After that, no more fixes. Most teams I talk to have the upgrade on a list somewhere, and most of them are worried about the data: pg_upgrade, extensions, downtime.

The thing I'd worry about is smaller and later. It's a login failure that doesn't happen on upgrade day.

What changed in 14

PostgreSQL 14 changed the default of password_encryption from md5 to scram-sha-256. Every new password is stored as a SCRAM verifier unless you change that setting.

Here's the part that makes it sneaky. pg_upgrade keeps the MD5 hashes you already have. So you upgrade a 12 or 13 server to 14 (or straight to 18), your apps reconnect, everything is green, and you close the ticket.

Then, weeks later, someone rotates the app's database password. The new one is stored as SCRAM. A Java service that has been fine for years starts failing with:

org.postgresql.util.PSQLException: The authentication type 10 is not supported.

Type 10 is SCRAM. The driver doesn't speak it. Moving with dump and restore doesn't avoid it: pg_dumpall --globals-only carries the MD5 hashes over as they are, so you're in the same position. It only fails on day one if you create the roles by hand on the new server, because then every password is set again.

Which pgjdbc versions can log in

pgjdbc added SCRAM-SHA-256 in 42.2.0, only in the Java 8+ build. But 42.2.0 shipped with a known issue: the SCRAM client classes weren't packaged, so it didn't actually work. 42.2.1 fixed that a week later.

So the floor is 42.2.1, and not a .jre6 or .jre7 build. Those older-Java builds never got SCRAM at any version. That catches people: a 42.2.x.jre7 jar looks newer than 42.2.1, and it still can't log in.

The same change hits other clients. The floors I check against, each taken from the client's own changelog or source:

Go: lib/pq 1.1.0, pgx 3.4.0
Node: pg 7.9.0
.NET: Npgsql 3.2.7
Anything built on libpq: libpq 10
PgBouncer: 1.11, or 1.14 if it looks passwords up with auth_query or keeps SCRAM secrets in auth_file

How to check before you upgrade (all read-only)

1. Find the PostgreSQL 14 servers in your cluster.

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.containers[*]}{.image}{" "}{end}{"\n"}{end}' \
  | grep -E 'postgres[^ ]*:14'

If an operator (CloudNativePG, Crunchy, Zalando) runs the database, check the image in its cluster resource as well.

2. See how each login role's password is stored today. As a superuser:

SHOW password_encryption;

SELECT rolname,
       CASE WHEN rolpassword LIKE 'md5%'           THEN 'md5'
            WHEN rolpassword LIKE 'SCRAM-SHA-256%' THEN 'scram'
            ELSE 'none' END AS stored_as
FROM pg_authid
WHERE rolcanlogin;

Every md5 row is a password that will turn into SCRAM the next time someone resets it.

3. Find the driver each Java service actually ships. Not the version in a pom from two years ago, the jar in the running image:

kubectl exec -n <ns> <pod> -- sh -c 'find / -name "postgresql-*.jar" 2>/dev/null'

Anything below 42.2.1, or with .jre6/.jre7 in the name, gets bumped before the upgrade, not after the first failed rotation.

Why this one is worth an hour this week

The end-of-life date is a deadline you can see. This failure is one you can't: the upgrade passes, the dashboards stay green, and it surfaces on whatever day a password rotates. Usually not a day anyone connects with the upgrade.

Checking takes three queries. Finding it during an incident takes a lot longer.

I read the Kubernetes changelogs from 1.32 to 1.37 properly

Silpa Nallagopula — Tue, 15 Sep 2026 13:04:57 +0000

I've been building an upgrade-readiness tool for about a year, and for most of that time I worked from the changelogs the way everyone does: find the version, skim "Urgent Upgrade Notes", move on. Then a cluster I care about hit something on 1.33 that I hadn't seen coming, and I decided to stop skimming.

So I cloned kubernetes/kubernetes and wrote a script that pulls out, per release, the sections that carry obligations: Urgent Upgrade Notes, Action Required, Deprecations, API Changes, Known Issues. Patch releases included, because they sneak things in. That's 36 files going back to 1.2. Then I sat down with 1.32 through 1.37 and read them. All of it. It took most of a weekend and I don't recommend it as a weekend.

The numbers
152 bullets that require someone to do something. Of those, roughly 73 I can verify by looking at the cluster: objects, CRDs, ConfigMaps, node info. The other 79 or so I can't, because they're about kubelet flags, node-level config, scrape configs, or client-side kubeconfigs. For the 1.33 → 1.34 hop specifically, 7 of the 10 urgent items are in that second bucket. I'll come back to why that matters.

Things I didn't know
The changelog is wrong about locked feature gates, at least twice. When a gate gets locked to its new default, any cluster that had it set the other way changes behaviour on upgrade. The notes mention some of these. The actual source of truth is the LockToDefault field in kube_features.go, so I diffed that between tags instead of trusting the prose. 73 gates locked across 1.32–1.35. And two dates are wrong: MaxUnavailableStatefulSet is listed under 1.35 but locks in 1.37, and StatefulSetAutoDeletePVC is listed under 1.32 but locks in 1.33. I only caught them because the source diff and the notes didn't agree. If your rules come from the release notes, you have the same two errors I had.

1.30 has no Urgent Upgrade Notes. None. I re-checked because I assumed my extractor had broken. It hadn't. There just aren't any. A tool that reports something for every hop is making it up for this one.

The scariest bullets in 1.34 sit under the least scary heading. They're under a line that literally says "(No, really, you MUST read this)": apiserver metric-label renames, the kubelet --cloud-config flag removed, static-pod admission changed. But in the rendered file, the heading above that block is just "v1.34.0". I scrolled past it the first time. So did the LLM I was testing that week, which is a story for another post.

Old changelogs are formatted differently in ways that break naive parsing. The 1.7–1.15 era puts everything under a narrative H1, and 1.7 bold-wraps its headings. Nobody cares about this unless you write the extractor, and then you care a lot.

The add-ons move when Kubernetes moves. A 1.36 target drags CoreDNS and kube-proxy along, and Istio, Karpenter, cert-manager if you run them, each with their own notes. Istio's upgradeNotes cover maybe half of what actually breaks; the rest is in the Removed/Deprecated lists and in flags whose defaults flip. Istio also drops compatibility profiles without a release note. The 1.21 profile vanished at 1.25. I only noticed because I was diffing the registration code.

What I did with it
Rules. Hand-written, one per bullet, per version, with the upstream sentence quoted verbatim and a kubectl command to verify it. About 600 across Kubernetes and seven add-ons now, plus 146 locked-gate rules generated from the source diff rather than the changelog. They run in a small Go CLI that needs a kubeconfig context and never writes anything:

kube-upgrade-check --context my-cluster --target 1.36 -o json
Every rule comes back as APPLIES with the objects named, CLEARED, or NOT DETECTABLE with the manual check spelled out. That last outcome exists because of the 79. I'd rather the tool say "I can't see your kubelet flags, go run this" than pretend it looked.

Repo: github.com/runtimez-com/kube-upgrade-check
My first pass missed 31 items I only found on the second read, so I'm fairly sure there are more. If your last upgrade broke on something that isn't in there, the changelog line in an issue is enough.

From Kubernetes PR to Deployment Verdict: Approve, Warn, or Block

Silpa Nallagopula — Mon, 03 Aug 2026 04:00:00 +0000

A small change in a Helm values file can look completely harmless during review.

The YAML is valid.

The Helm chart renders successfully.

The CI pipeline is green.

The pull request gets approved.

But after deployment, the application starts failing, pods are restarted, or availability is reduced.

The problem is that most CI checks focus on whether the Kubernetes configuration is technically valid. They do not always explain what the change will do to the workloads already running in the cluster.

Why normal CI checks are not enough

Schema validation is useful. Policy checks are useful. Linting is useful.

But passing those checks does not automatically mean that a change is safe to deploy.

A pull request may contain changes such as:

Reducing replicas from three to one
Removing a memory limit
Changing a readiness probe
Removing a PodDisruptionBudget
Using a floating image tag
Modifying a NetworkPolicy
Changing scheduling or topology constraints

Every one of these changes may be valid Kubernetes configuration.

The operational risk depends on the application, the current cluster state, the traffic pattern, and the workloads affected by the change.

A simple example

Consider this change:

replicaCount: 3

changed to:

replicaCount: 1

There is nothing syntactically wrong with the new value.

The Helm chart will render, and Kubernetes will accept the deployment.

But the operational impact could be significant.

With only one replica:

A restart may temporarily make the service unavailable
A node drain could interrupt the application
A rolling deployment may have less capacity
A single pod failure could affect all users of the service

A reviewer should not have to discover all of this by manually reading the YAML and trying to understand the possible impact.

The pull request should clearly explain that availability is being reduced.

From validation to a deployment verdict

Instead of returning only a list of configuration changes, a PR-time check should provide a clear deployment verdict.

Approve

The change does not introduce a meaningful reliability, security, cost, or availability risk.

For example, updating a harmless label or increasing a safe resource limit may be approved automatically.

Approve with warning

The change is likely safe, but the reviewer should be aware of a possible impact.

For example, increasing memory limits may increase cost, even if it does not create an immediate reliability problem.

Require approval

The change introduces meaningful operational risk and should be reviewed by the appropriate owner.

Examples may include:

Reducing replicas
Modifying resource limits
Changing probes
Removing a PodDisruptionBudget
Changing network access rules

Block

The change violates a critical reliability, security, or organizational policy.

Examples may include:

Removing required resource limits
Introducing a known critical vulnerability
Using a prohibited image tag
Breaking a mandatory availability requirement
Blocking an active service connection

What the reviewer should see

A useful PR-time verdict should explain more than whether a rule passed or failed.

The reviewer should be able to understand:

What materially changed
Why the change matters
Which workloads or services are affected
What the possible blast radius is
Whether the change requires additional approval
Whether the change can be rolled back safely
What action should be taken before deployment

The goal is not to replace engineering judgment.

The goal is to give reviewers enough context to make a better decision before the change reaches production.

The larger idea

Kubernetes deployment safety is not only a YAML-validation problem.

It is a decision problem.

Teams need to know whether a change should be approved, approved with a warning, reviewed by an owner, or blocked before it reaches the cluster.

That decision should be based on the operational effect of the change, not only whether the configuration is valid.

Originally published on the Runtimez blog:

https://runtimez.io/blog/pr-time-kubernetes-verdict