Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
🛠️ Pipelines in the Wild #5
⚡ Byte Size Summary
- Understand why a Helm sub-chart with a hardcoded deprecated API can pass CI, deploy cleanly, and still crash production ten minutes later
- See why admission-time governance and a managed control plane solve two completely different problems — and why conflating them is how teams get burned
- Walk away with a clearer sense of where platform ownership should actually sit as an organization scales
The Story
I was building an optimization model for scheduling — an internal Red Hat project, nothing exotic. Somewhere in the middle of it I noticed something uncomfortable: I wasn't spending my time on the application anymore. I was spending it on the platform underneath it. Roughly half my working hours were going into upgrade cycles — chasing down which APIs had quietly disappeared and which of my dependencies would break the moment the Kubernetes cluster moved forward a minor version.
The cluster in question was a large, multi-tenant OpenShift environment shared across several engineering teams. We didn't control what those teams deployed. Each had its own CI/CD pipeline, its own release cadence, its own tolerance for risk.
The failure that made this real happened during what should have been a routine minor version upgrade. Our static Git repositories looked clean — we'd grepped them, audited them, signed off. What we hadn't accounted for was a third-party Helm chart with a hardcoded, deprecated API version buried inside a sub-chart, several layers deep. It didn't show up in any static scan because it was never in the Git source in a form our tooling could see — it was rendered dynamically, at deploy time, by Helm's templating engine.
CI passed. The deploy succeeded. Ten minutes later, production pods started crashing — silently, with no pipeline failure and no obvious alert pointing at the cause. The control plane could no longer parse the schema for a resource that, as far as our source control was concerned, didn't exist.
That was the moment the platform stopped being something I occasionally maintained and became something that was actively costing me the ability to do my actual job.
The Problem
Platform engineers running multi-tenant OpenShift clusters feel this pain directly, and it isn't really about Kubernetes deprecating APIs — that's expected, documented, and version-scheduled. The real problem is that you cannot audit what you don't control, and in a shared cluster, tenant teams control their own manifests, their own Helm charts, and often their own third-party dependencies.
Static auditing — grepping Git repos for deprecated API references before an upgrade — assumes the thing you're deploying is fully visible in source control. It isn't, once Helm templating, mutating webhooks, or dynamic chart composition enter the picture. The cost isn't abstract. It's engineering time spent firefighting instead of building. It's a 48-hour change-control freeze window that becomes meaningless if something breaks anyway. And it's production incidents that look like mysterious pod crashes rather than the predictable, schedulable event an API deprecation actually is.
Why Existing Approaches Fall Short
We tried three things before landing on what actually worked, and each one failed against a specific constraint rather than being generically "not good enough."
A dedicated pre-prod cluster running the target version early. This sounds like the obvious answer, and it solved nothing for us. With a dozen independent teams deploying at will, a team could pass validation against pre-prod on Monday and push a completely different, unvalidated Helm chart to production on Wednesday — squarely inside the 48-hour freeze window. Pre-prod testing has no visibility into what changes between the test and the deploy.
Moving to a managed platform as the fix. Red Hat OpenShift Service on AWS (ROSA) and Azure Red Hat OpenShift (ARO) genuinely solve control-plane toil — patching, minor version upgrades, master node lifecycle — by shifting that operational burden to the provider's SRE team. That's real, and it matters, but it does not solve this problem. If a tenant deploys a deprecated resource, a managed platform will still drop it on schedule during the upgrade, exactly like a self-managed cluster would. You'd be paying a premium for control-plane operations and still hitting the same silent runtime failure, because managed platforms don't rewrite your application manifests. They manage the masters, not the tenants.
CI-only static scanning against Git. This is what we tried first, and it's what failed in the incident above. These tools only see what's committed. They're blind to Helm templates that render dynamically, mutating webhooks that alter objects after submission, and direct overrides developers reach for during incidents. The deprecated API in our failure never existed in a form static scanning could catch.
What all three approaches share is a gap between what we could inspect and what actually got submitted to the cluster. Closing that gap meant moving enforcement to the one place that sees every object regardless of where it came from: the moment it's admitted, on the live cluster.
The Architecture
We adopted OPA Gatekeeper as an admission-time gate — a control that sits between anything being submitted to the cluster and the API server itself, evaluating the object as it actually arrives rather than as it appears in Git. That distinction is the whole point. It doesn't matter whether a manifest came from a clean pull request, a Helm sub-chart three layers deep, or a developer running a command by hand during an incident — everything crosses the same checkpoint before it's persisted.
We deliberately didn't configure this as one global switch. Application namespaces were set to fail open — if the admission control itself became unreachable, deploys would still go through rather than grinding every tenant to a halt. The kube-system namespace was set to fail closed, because a control-plane-critical failure was a risk we were willing to accept blocked deploys to avoid. That split existed because a single global policy in either direction was wrong for at least one half of the cluster.
The decision above took us longer to get right than the technical architecture did: separating who manages the control plane from who governs what gets deployed onto it. A managed platform like ROSA or ARO sits entirely on the control-plane side of that line. Admission governance sits on the other side, and it doesn't move just because you change who's patching the masters.
How It Works
The mechanism itself is simple to describe even though getting it right operationally wasn't. Every object submitted to the cluster — however it arrived — passes through the admission gate before the API server accepts it. If the object matches something on our list of deprecated or removed APIs, it's rejected on the spot, and the team that submitted it gets a clear, immediate failure instead of a clean deploy that quietly breaks ten minutes later.
Running this in production for a while surfaced something we didn't expect. A handful of teams found a way around it — not maliciously, just by submitting resources against an older version of the same API. Kubernetes itself has a conversion mechanism that will silently upgrade an older-format object after it's been admitted, and our check was evaluating the object as submitted, not as it ended up after conversion. The check passed on a value that was about to be rewritten underneath it.
We caught this with a second, entirely separate tool — kubent — running on a schedule against the live state of the cluster, not against Git and not as part of the admission path. It doesn't prevent anything. It tells you, after the fact, what got through anyway. That pairing turned out to matter more than either control on its own: one thing stopping bad objects at the door, and something completely independent double-checking what actually ended up living in the cluster.
Security and Operational Considerations
Fail-open is a real availability decision, and it comes with a real cost. Because application namespaces are configured to let requests through if the admission control is unreachable, that unreachability becomes something an attacker — or just a badly behaved workload — could try to trigger deliberately. Flood the admission control with large enough or complex enough requests, push it past its timeout, and for a window, everything gets admitted unvalidated. That's not a hypothetical. It's a direct consequence of choosing availability over strict enforcement, and it's the tradeoff you're accepting the moment you set that policy.
The string-matching gap described above is the other side of the same coin. The admission check was doing exactly what it was configured to do — it just wasn't looking at the version of the object that eventually mattered. That's not a bug in the tool. It's a mismatch between what got inspected and what got persisted, and it's the kind of gap that only shows up once you're running this against real, adversarial-enough traffic in a large, shared environment.
Whatever runs the admission check needs broad read access across the cluster to do its job, which means it's also a meaningful blast radius if it's ever compromised. That's not a reason not to run it. It's a reason to treat changes to its configuration with the same scrutiny you'd apply to any other cluster-wide permission change, not as routine tooling maintenance.
What Breaks at Scale
A few things stopped working cleanly as this scaled past our original setup, and none of them were the technology itself.
The notifications became noise. Every denial generated an alert in a shared channel, and once teams saw enough of them, they stopped reading the channel at all — which meant the alerts that actually mattered started getting missed along with everything else. That's not a tooling failure. It's what happens to any high-volume signal that isn't triaged deliberately.
The deny-list itself became a maintenance burden once we were running more than a handful of clusters. Kubernetes deprecates APIs on a rolling schedule across minor versions, and once different clusters are sitting on different versions — which happens naturally when every business unit has its own freeze window — a single list can't be correct everywhere at once. What's deprecated on one cluster might still be perfectly valid on another.
And the namespace classification that the whole fail-open/fail-closed split depended on started drifting. New platform-critical components — new operators, new ingress layers — landed in namespaces that weren't kube-system, and nobody went back and reclassified them. They silently inherited the fail-open behavior meant for ordinary application workloads, which defeats the entire point of the split. A one-time, manually maintained classification doesn't hold up as an organization keeps growing.
If we were building this again today, we'd move away from the string-matching approach entirely and toward something that evaluates the actual resolved structure of an object rather than a single field that can be silently rewritten out from under the check. That closes the exact gap kubent had to compensate for, by design, instead of needing a second tool to catch what the first one missed.
There's a broader point in all of this that's easy to miss until you've lived through it: putting an admission control in front of your cluster doesn't build a wall around it. It inserts a live, synchronous dependency directly into the control plane's request path. Once that exists, the health of your cluster isn't just a function of etcd and the API server anymore. It's also a function of how quickly that dependency responds.
Quick Recap
- Static auditing has a real blind spot — anything rendered at deploy time, rather than committed to Git, is invisible to source-based scanning no matter how disciplined the process
- A managed control plane and admission governance solve different problems — offloading upgrade toil to ROSA or ARO doesn't remove the need to govern what tenants actually deploy, and the two decisions shouldn't be conflated
- Prevention and detection work best as separate layers — an admission-time check can have real gaps, and a second, independent tool watching live cluster state is often what actually catches them
GitHub Repo
This article documents an internal governance decision rather than a shareable implementation — there's no companion repository for this one.
What's Next?
If the alert-fatigue problem in this article sounds familiar, the next stop in the series is worth a read: Retry Logic and Tiered Alerting in GitHub Actions — a closer look at the same "signal drowning in noise" problem, this time from the CI/CD pipeline side rather than the admission-control side.
Written by Pipeline & Prompts | Byte size guides on DevOps, Cloud and AI
Top comments (0)