Early in my career I worked somewhere that deployed once a month. It felt safe on paper, a big careful release with lots of testing behind it. In practice each deploy was terrifying, because it bundled a month of changes from a dozen people, and when something broke, which it always did, we had no idea which of the hundred changes was the culprit. We would spend the night bisecting a giant diff under pressure, and the "safe" monthly cadence was the reason the nights were so bad.
The counterintuitive fix was to deploy far more often, in much smaller pieces. It sounds riskier and it is the opposite. When you ship one small change and something breaks, the cause is almost always obvious, because there is only one thing it could be. The rollback is trivial, because you are reverting a single well-understood change instead of untangling a month of interleaved work. Small batches turn debugging from an investigation into a glance.
The reason big deploys feel safe is a trick of perception. Batching changes reduces how often you feel fear, so it feels like less risk. But it concentrates all of that risk into rare, high-stakes events where the blast radius is enormous and the diagnosis is hard. You are not removing risk, you are saving it up and detonating it all at once. Frequent small deploys spread the same risk into pieces small enough to handle calmly.
None of this works without the machinery underneath it. Shipping small and often is only sane if your pipeline is fast, your tests are trustworthy, and rolling back is a routine, boring operation rather than an emergency procedure someone half-remembers. That investment is the actual point of a deployment pipeline. It exists to make each individual change cheap and reversible, so that shipping stops being a dramatic event.
The habit I try to instill on every team is to keep the unit of change small enough that reverting it is never scary. If the thought of rolling back a deploy makes people nervous, the deploy was too big. Small changes, shipped often, with a boring rollback: that is what "move fast" actually looks like when you also intend to stay up.
– Serguey Shinder
Top comments (0)