A small change in a Helm values file can look completely harmless during review.
The YAML is valid.
The Helm chart renders successfully.
The CI pipeline is green.
The pull request gets approved.
But after deployment, the application starts failing, pods are restarted, or availability is reduced.
The problem is that most CI checks focus on whether the Kubernetes configuration is technically valid. They do not always explain what the change will do to the workloads already running in the cluster.
Why normal CI checks are not enough
Schema validation is useful. Policy checks are useful. Linting is useful.
But passing those checks does not automatically mean that a change is safe to deploy.
A pull request may contain changes such as:
- Reducing replicas from three to one
- Removing a memory limit
- Changing a readiness probe
- Removing a PodDisruptionBudget
- Using a floating image tag
- Modifying a NetworkPolicy
- Changing scheduling or topology constraints
Every one of these changes may be valid Kubernetes configuration.
The operational risk depends on the application, the current cluster state, the traffic pattern, and the workloads affected by the change.
A simple example
Consider this change:
replicaCount: 3
changed to:
replicaCount: 1
There is nothing syntactically wrong with the new value.
The Helm chart will render, and Kubernetes will accept the deployment.
But the operational impact could be significant.
With only one replica:
- A restart may temporarily make the service unavailable
- A node drain could interrupt the application
- A rolling deployment may have less capacity
- A single pod failure could affect all users of the service
A reviewer should not have to discover all of this by manually reading the YAML and trying to understand the possible impact.
The pull request should clearly explain that availability is being reduced.
From validation to a deployment verdict
Instead of returning only a list of configuration changes, a PR-time check should provide a clear deployment verdict.
Approve
The change does not introduce a meaningful reliability, security, cost, or availability risk.
For example, updating a harmless label or increasing a safe resource limit may be approved automatically.
Approve with warning
The change is likely safe, but the reviewer should be aware of a possible impact.
For example, increasing memory limits may increase cost, even if it does not create an immediate reliability problem.
Require approval
The change introduces meaningful operational risk and should be reviewed by the appropriate owner.
Examples may include:
- Reducing replicas
- Modifying resource limits
- Changing probes
- Removing a PodDisruptionBudget
- Changing network access rules
Block
The change violates a critical reliability, security, or organizational policy.
Examples may include:
- Removing required resource limits
- Introducing a known critical vulnerability
- Using a prohibited image tag
- Breaking a mandatory availability requirement
- Blocking an active service connection
What the reviewer should see
A useful PR-time verdict should explain more than whether a rule passed or failed.
The reviewer should be able to understand:
- What materially changed
- Why the change matters
- Which workloads or services are affected
- What the possible blast radius is
- Whether the change requires additional approval
- Whether the change can be rolled back safely
- What action should be taken before deployment
The goal is not to replace engineering judgment.
The goal is to give reviewers enough context to make a better decision before the change reaches production.
The larger idea
Kubernetes deployment safety is not only a YAML-validation problem.
It is a decision problem.
Teams need to know whether a change should be approved, approved with a warning, reviewed by an owner, or blocked before it reaches the cluster.
That decision should be based on the operational effect of the change, not only whether the configuration is valid.
Originally published on the Runtimez blog:
Top comments (0)