At half past eight on a Friday evening the mobile API started returning errors on a particular screen. The cause was a missing null check, found in about twenty minutes. An engineer branched, wrote one line, got a review, merged, and our pipeline deployed main to production the way it always does. The errors stopped. We closed the incident at ten past nine and were quietly pleased with ourselves.
The second wave started an hour later and was much larger than the first.
Production had been running the same build for three weeks. The release train had been held while a partner integration finished, which was a deliberate, reasonable decision taken in a meeting I was in. Main had carried on moving. So when we deployed main to fix one line, we also shipped a hundred and forty other commits, including a change to how tokens are validated and a new required header on an internal call, at nine o'clock on a Friday, with no soak, no announcement and nobody watching except the three of us on the incident.
Every single one of those commits was reviewed, tested and merged by people doing their jobs correctly. What nobody did was look at how far production had fallen behind, because there was nothing anywhere that showed it. Our pipeline says deploying main. It does not say deploying main, which is a hundred and forty commits and four schema changes ahead of what you are running.
The deploy step now prints that. Commits ahead, services touched, migrations included, and the names of anyone whose work is going out. Above twenty commits it requires a second approval, which has already stopped one repeat. Hotfixes branch from the tag that is actually running and get cherry picked forward, rather than being cut from main, which is a five minute difference in effort and a completely different blast radius.
And we stopped holding trains. Main deploys daily whether or not anybody calls it a release, so the distance between what is running and what is merged is never more than a day. The partner integration that we had held the train for was not even affected by any of it.
A deploy is not the size of your diff. It is the distance between what is running and what you are about to run, and we had never once measured that distance.
– Sergey Shinder
Top comments (0)