There is a category of learning almost nobody uses on purpose, and it is sitting right there in your company wiki.
Other people's incident writeups.
We treat postmortems as paperwork. The team that had the outage writes it, a few people skim it, it gets filed, and everyone moves on until the next one. Meanwhile it is one of the richest pieces of technical writing your organisation produces.
Think about what is actually in one.
A real system, described honestly, under stress. The assumption somebody made that turned out to be wrong. The gap between how the architecture was drawn and how it behaved at three in the morning. The monitoring that did not fire. The fix, and the reason the obvious fix was not the right one.
You cannot buy that. No course contains it, because it is specific to systems that actually exist and people who actually made decisions under pressure.
So go read them. Not just yours.
Pick one a week from a team you do not work on. Read it before the resolution section and ask yourself what you would have checked first. Then read the rest and see where your instincts were off.
That gap is the lesson. You are getting the experience of an outage without the outage.
Public ones work too. Plenty of companies publish theirs, and the good ones read like detective stories. You learn how distributed systems fail, how retries turn a small problem into a large one, how a config change ships faster than anyone can review it.
And you learn the cultural part, which matters just as much. You see what a blameless writeup sounds like compared to one hunting for a person to pin it on. You will need to know the difference when it is your turn to write one.
Most learning advice tells you to build more things. That is good advice.
But you will also learn a great deal from watching other people's things break, and from asking, before you read the answer, what you would have done.
– Asael Shinder
Top comments (0)