Every engineering team has a version of this story: A big release is coming. Everyone knows it. The feature is complex, the codebase changes are significant, and there has been a lot of discussion about the risks. So the team does everything right. Extra testing. Careful code review. Staged rollout. Someone watching the dashboards during deployment. Post-deployment monitoring for the first few hours.
The big release goes perfectly.
Three days later, a routine dependency update that nobody thought twice about takes down a critical service for two hours.
This is not bad luck. It is a pattern that appears so consistently across engineering teams that once you see it you cannot unsee it.
The deployments that get scrutinised rarely cause the incidents. The deployments that cause incidents are usually the ones that felt safe enough not to scrutinise.
Why Attention Is Not Evenly Distributed
Engineering teams do not treat all deployments equally and they should not. A major feature release carrying two weeks of work across multiple services deserves more scrutiny than a one-line config change. Allocating attention based on perceived risk is rational.
The problem is that perceived risk and actual risk are not the same thing.
Perceived risk is based on what the team knows. The size of the change, the complexity of the code, the areas of the system that were modified. These are visible signals. They are easy to evaluate and easy to use as a basis for deciding how much testing and monitoring a deployment needs.
Actual risk includes all of that plus everything the team does not know. The dependency that changed behavior in a way nobody noticed. The integration point that was sensitive to a change in a completely different service. The edge case that only appears under specific production conditions that staging never replicates.
The deployments that get scrutinised are the ones where perceived risk is high. The deployments that cause incidents are often the ones where actual risk was higher than perceived risk. And actual risk almost always concentrates in places the team was not looking.
The Specific Failure Mode
The deployment that breaks things without anyone watching tends to follow a specific pattern.
A change gets merged that touches something the team considers low risk. A dependency version bump. A configuration update. A small refactor to an internal utility. Something that has been done dozens of times before without incident.
What nobody knows is that this particular change has a side effect at an integration boundary. A downstream service that this code calls -- or that calls this code -- has changed its behavior since the last time anyone looked carefully at that integration. The new deployment interacts with the changed downstream service in a way that produces a failure.
The test suite does not catch it because the tests for this integration are running against mocks that reflect how the downstream service behaved several months ago. The staging environment does not catch it because the downstream service in staging has not been updated to match production. The deployment completes successfully. The failure appears hours later when a specific production workflow hits the broken integration point.
What Actually Determines Deployment Risk
The missing variable in most deployment risk assessments is the accuracy of the testing infrastructure, not the size of the change.
A large complex change deployed against an accurate, well-maintained test suite that reflects current service behavior is less risky than a small simple change deployed against a test suite running on stale mocks and outdated integration assumptions.
The big release that got all the attention was tested carefully against the current state of the system. That is why it went well. The routine update that caused the incident was deployed against testing infrastructure that had quietly drifted from production reality. That is why it failed.
This reframes what good software deployment practice actually looks like. It is not about scaling scrutiny to the size of the change. It is about maintaining testing infrastructure that makes the actual risk visible regardless of how the perceived risk looks.
The Uncomfortable Implication
If deployment incidents concentrate in changes that felt safe rather than changes that looked risky, then adding more scrutiny to big releases is not the primary lever for reducing incidents.
The primary lever is keeping the test suite accurate enough that low-scrutiny deployments are actually low risk rather than just appearing that way.
That means integration tests that reflect current service behavior rather than behavior from six months ago. Mocks that are derived from real production interactions rather than developer assumptions about how dependencies behave. Pipeline stages that catch behavioral regressions before merging rather than discovering them in production.
This is not a new idea. Most engineers know that test accuracy matters. The reason it does not get addressed is that stale mocks and drifted integration tests do not announce themselves. The tests keep passing. The dashboards stay green. The problem only becomes visible when a deployment that nobody was watching breaks something that should have been caught.
What to Actually Watch
The thing worth watching is not the deployment. It is the gap between what the test suite is validating and what the production system is actually doing.
That gap grows silently. Every time a downstream service deploys and the corresponding mocks do not get updated, the gap widens. Every time an integration changes and the tests for that integration do not follow, the gap widens. The gap does not announce itself until a deployment falls into it.
The deployments that break things are not the ones that looked risky. They are the ones that fell into a gap that had been accumulating for months without anyone noticing.
Closing the gap is harder than watching the dashboards during a big release. It is also the only thing that actually works.
Top comments (0)