Every team eventually builds the same mental shortcut: green pipeline means it's safe to ship. Tests passed, the build compiled, the checks are all checkmarks. Merge, deploy, move on.
That shortcut is wrong more often than most postmortems admit, and it's wrong in a specific, repeatable way: CI/CD pipelines are very good at verifying that code does what it's supposed to do, and almost silent about whether the environment it's about to land in still matches the one it was tested against.
Three places this gap actually bites, in order of how often I've seen each one cause a real incident.
1. The test suite passed against a database that no longer looks like production
Migrations are the classic case. A pipeline runs unit and integration tests against a freshly-migrated test database, everything's green, and the deploy goes out. What the pipeline didn't check: whether that migration is actually safe to run against a production table with real row counts, real lock contention, and real traffic hitting it mid-migration.
A migration that adds a NOT NULL column with no default locks the table for the duration of the backfill on most engines. On a test database with a thousand rows, that's invisible — it finishes before you'd notice. On a production table with tens of millions of rows, it can hold a lock long enough to back up every write for minutes, which looks nothing like "the tests failed" and everything like "the site is down."
The fix isn't a smarter test — it's a different question in the pipeline: not "does this migration produce the right schema," but "is this migration safe to run online, at this table's actual size, without an explicit maintenance window." That's a check most CI setups never encode, because it requires knowing something about production the pipeline usually isn't told.
2. The deploy succeeded, but nobody ever tested the rollback
Almost every deploy pipeline is optimized to answer one question: did the new version start successfully? Very few are set up to actually exercise the answer to "if this version is bad, can we get back to the old one without a human improvising at 2am?"
Rollback usually gets tested for the first time during an actual incident, which is the worst possible time to discover that the previous image was already garbage-collected off the registry, or that a schema migration in the new release isn't backward-compatible with the old code, or that a feature flag the new version depends on doesn't exist in the old one's code path.
If a pipeline can deploy a version, it should also be able to prove it can un-deploy that version, on a schedule, not just in theory. That's a genuinely different pipeline stage than "ship the new thing" — most teams have the first and skip the second.
3. Staging is green because staging quietly drifted from what production actually runs
This is the slow one. Staging starts as an accurate mirror of production. Six months later, staging is running a slightly older version of a managed service, a smaller instance class, a different set of feature flags, or a mocked-out third-party dependency that production talks to for real. Every pipeline run against staging is honestly green — the code really does work, in an environment that no longer represents what it's about to be deployed into.
Nobody decides to let this happen. It accumulates one small, individually-reasonable shortcut at a time: "let's just mock that API in staging for now," "we'll upgrade staging's database version next sprint." Each shortcut is fine in isolation. The sum of them is a green pipeline that's answering a question nobody's actually asking anymore.
The only real defense is treating environment parity as something that gets checked, not assumed — a periodic, automated diff between staging's actual configuration and production's, not a one-time setup step everyone trusts forever after.
The pattern underneath all three
None of these are exotic failures. They're the ordinary, boring result of a pipeline that was built to answer "does the code work" being trusted to also answer "is it safe to put this in front of real traffic right now" — a question it was never actually wired to check.
Closing that gap isn't about adding more tests in the traditional sense. It's about adding a small number of specific, deliberate checks that ask about the deploy itself, not just the code: is this migration safe at production scale, is rollback actually exercised, does staging still resemble what's about to receive this change. Each one is cheap to build once someone decides it's a gap worth closing — the hard part is noticing the gap exists before an incident points it out.
I do this kind of infrastructure/CI-CD hardening work professionally through CyberTechnology — happy to talk through a specific pipeline if any of this sounds familiar.
Top comments (0)