Most outages I have looked into did not start with a bad line of code. They started months earlier, with a decision that made sense at the time and was never written down.
Someone chose a timeout, a retry policy or a data format for a good reason. The reason lived in one person's head. Then that person moved on, the system changed around the decision, and nobody knew it was still holding something up.
What a written decision needs
It does not need to be long. Five lines are enough:
- What we decided. One sentence.
- Why. The constraint or problem that forced it.
- What we gave up. The option we did not take.
- When to revisit. A date, a traffic level, or an event.
- Who decided. A name, not a team.
Where to keep it
Next to the code or the system it affects, not in a slide deck. If an engineer changing the timeout cannot find the reason in under a minute, the record might as well not exist.
The habit, not the template
Teams often start with a template and stop after three entries. The habit matters more than the format. A good test: in every design review, ask "where will this be written down?" before anyone leaves the room.
What it buys you
When something breaks at 2 a.m., the on-call engineer can see why the system works the way it does, and whether the old reason still holds. That is often the difference between a fix and a second outage.
I'm Pratyush Shivam, CTO at Konfidio in Berlin, where we build TrustedTrucks. More about my work at pratyushshivam.com.
Top comments (0)