It had passed in development. It had passed in the integration environment. It had passed in the staging environment that we described, in writing, as production-like. Then it went to production at half past seven in the evening and failed within ninety seconds, on a configuration difference that had been sitting there since before anyone on the team had joined. We spent the next two hours not fixing a bug in the release, but discovering what our own production environment actually contained, which is a humbling way to spend an evening.
Environments drift because they are maintained by different pressures. Production gets emergency changes made at speed by people solving a real problem, and those changes are honestly intended to be replicated downward and frequently are not. The lower environments get shortcuts, because they are cheaper and nobody is watching: a smaller instance here, a mocked dependency there, a certificate that expired and was replaced with something more convenient. Each individual divergence is defensible. The accumulated difference is a staging environment that tests the code perfectly and the deployment not at all.
The dangerous part is the false confidence, which is worse than no confidence at all. If we had known staging was unrepresentative, we would have released carefully, watched closely, and had a rollback ready. Instead we had a green pipeline and a comfortable team, so we released the way you release when a machine has already told you it is fine. The automation had not lied. It had answered exactly the question we asked it, which was whether the change worked in staging, and we had heard it answer a different question entirely.
What changed afterwards was less glamorous than a new tool. Both environments are now built from the same definitions, and the differences that must exist are listed explicitly in one file that a human reviews. A scheduled job compares the two and reports the drift, so divergence becomes a thing we see weekly rather than a thing we discover during an incident. And when a fix goes into production out of hours, the ticket is not closed until it exists everywhere else too.
A test environment is only evidence to the extent that it resembles the place you are actually deploying.
– Serguey Shinder
Top comments (0)