Break It on Purpose Before It Breaks on Its Own

You learn more about a system in the twenty minutes it is failing than in the six months it behaves. The problem is that real failures arrive at four in the morning, in front of an audience, when you have no time to be curious.

So arrange some failures yourself, somewhere it does not matter.

Take a service you work on, put it in a local or scratch environment, and go looking for the edges. Stop the database and watch what your code does with the connection. Fill the disk. Make the dependency return a five hundred, then make it return nothing at all and just hang. Kill the process halfway through a write. Move the clock forward a day. Revoke the token. Send the endpoint an empty list, then the same request twice.

The discipline that makes this learning rather than poking is writing your prediction down first. One line. I think it will retry three times and then return a clear error. Then run it and see. Where the prediction and the result differ is exactly the part of the system you did not actually understand, and now you do, cheaply.

You will also learn your own tooling, which is the quiet half of the value. Where the logs go when things are ugly. What the failure actually looks like in the dashboard, as opposed to how you imagine it looks. How long it takes you to notice. How long recovery takes when you are calm.

A few boundaries, because this is a habit that can go badly. Do it in an environment you are allowed to destroy. Tell your team what you are doing so nobody debugs your experiment. Never let it turn into a justification for trying things in production because that is where it is interesting.

Then leave a trace. A short note listing what you broke, what you expected, what really happened. Six months later, when the same thing happens for real, that note is the difference between recognising it and rediscovering it under pressure.

The first genuine incident of your career is going to be frightening. It is a great deal less frightening if the failure in front of you is one you have already met on a quiet afternoon.

– Asael Shinder