Something is down. The engineers who know the system best are deep in logs, and somebody in the incident channel says, can you handle the updates? Sales are asking, support is taking calls, and a director has sent a message that just says status?
This is one of the most useful jobs in an incident, and it often goes to whoever happens to be nearest. Do it well and you take a great deal of pressure off the people fixing the problem.
Start by setting a clock. Tell everyone when they will hear from you next, and keep to it. Next update at half past two. People can wait a surprisingly long time when they know when news is coming. What they cannot bear is silence with no end. If nothing has changed by half past two, say so. No change, still investigating, next update at three is a real update.
Write each one in the same order. What is affected, in the reader's terms rather than yours. Customers cannot complete checkout, not elevated error rates on the payments service. What is being done. What you do not know yet. When the next update comes.
Be plain about what you do not know. Do not guess at a cause or a fix time to make the message sound better. A wrong estimate is remembered much longer than a missing one, and it gets repeated to customers by people who trusted you. We do not have an estimate yet is honest and professional.
Protect the people doing the fixing. When questions arrive for the engineers, collect them and answer what you can yourself. Pass on only what they truly need. Every interruption to the person reading the logs adds minutes to the outage.
Keep the audiences apart where you can. The engineers need detail in one channel. Support and sales need a short version they can repeat word for word. Customers need a plainer version again, usually approved by somebody before it goes out.
When it is over, send one final message that says so, what users might still notice, and when the full write up will arrive. Then keep every update you sent. Read back in order, they are often the clearest timeline anyone has when the review begins.
Monday: ask your team who writes the updates in your next incident, and offer to be that person once.
– Asael Shinder
Top comments (0)