The worst outage of my career was caused by a one-line change that three people, including me, described as trivial before we shipped it. We were changing a timeout value from thirty seconds to sixty. That is it. No new code, no schema change, nothing anyone would flag in review. It took down order processing for two hours, because that longer timeout let connections pile up faster than the pool could recycle them, and the pool exhaustion cascaded into everything that shared it.
Ever since, I have been suspicious of the word "trivial." Not because small changes are secretly dangerous, but because the label itself is the danger. When we call a change trivial, we give ourselves permission to skip the things that catch problems. We deploy it on a Friday afternoon. We do not watch it after it ships. We do not think about what depends on the thing we touched. The size of the diff tells you nothing about the size of the blast radius.
Real change discipline is not about slowing everything down with heavy process. Most teams that add ceremony to every change just teach people to route around it. What actually helps is a short, honest pause before any change that touches shared state, production configuration, or anything in the critical path: what breaks if this is wrong, how would we know within five minutes, and how do we undo it. If those three answers are quick and confident, ship it. If any of them makes you hesitate, the change was never trivial.
I also changed when we ship. Not because Fridays are cursed, but because the cost of a change is not just the change, it is your capacity to respond when it goes wrong. A "trivial" change at 4pm Friday with a skeleton crew is a much bigger risk than the same change at 10am Tuesday with the whole team awake.
The lesson stuck: judge a change by its blast radius and your ability to recover, never by the length of the diff. The system does not care how small your edit looked in the pull request.
– Serguey Shinder
Top comments (0)