The incident commander role belongs to the person who keeps the big picture during an outage, hands out tasks and makes decisions while others hunt for the bug. Large companies have on-call rotations and whole SRE departments for this. Small teams usually have one chat channel where twelve people suddenly type at once, plus a team lead everyone expects to “just sort it out”.
I care about this topic because team leads in particular are caught in the middle during an outage: they are supposed to coordinate, yet often would much rather dig into the logs themselves. After my article on the blameless postmortem, the obvious follow-up question was: what happens before that, while things are on fire? That is what this post covers. You get a lean role model for teams of two to eight people, the incident commander’s tasks step by step, templates for status updates and a handover, and a printable cheat sheet.
In short
An outage runs more calmly when exactly one person coordinates and does not fix things personally. In small teams anyone can take on this role, as long as it is assigned early and out loud. The most common pitfall is the team lead who automatically becomes incident commander and then starts debugging anyway.
What you will learn in this article
- Which four roles an incident needs and how to combine them in a small team
- How to tell that you should officially declare an incident
- What an incident commander does, in which order, and what not
- What status updates and a clean handover look like
- How your team can practice the role without a real outage
What is the incident commander role?
An incident is an unplanned disruption that noticeably affects users or threatens to: the API stops responding, payments hang, a queue fills up. The incident commander (IC for short) coordinates the response. They keep track of the overall picture, assign work, make decisions and make sure everyone involved knows where things stand.
The idea does not come from IT. In the 1970s, US fire services developed the Incident Command System because during large wildfires too many units were running around without clear leadership. Google adopted the principle for operating software and described it in the SRE book, and PagerDuty has published its entire incident response documentation openly. You will find both sources linked below.
Why the role has nothing to do with hierarchy
The most important point first: the incident commander role is temporary, not a job title. During the incident, this person has the final say on how the response runs, even towards their own manager. Afterwards they are simply part of the team again. PagerDuty is very explicit about it: on an incident, the IC is the highest-ranking person, no matter their everyday position. That sounds like a shift of power, but above all it takes pressure off. When people know who decides, nobody has to keep asking or covering themselves.
By the way, deep technical knowledge is not required. An IC needs a rough understanding of the system, good listening skills and clear speech. Others do the detailed work.
The four roles in an incident
Atlassian lists eight roles in its incident handbook, from social media lead to problem manager. For a team of five, that is absurd. In my view four roles are enough, based on the model from the Google SRE book:
| Role | Task | Does not |
|---|---|---|
| Incident commander | Coordinates, decides, assigns tasks, keeps the current picture up to date | Fix things, read logs, click through dashboards |
| Technical lead | Looks for the cause, proposes actions, carries out changes (alone or with experts) | Inform stakeholders |
| Communication | Writes status updates internally and externally, answers questions | Make technical decisions |
| Scribe | Records times, decisions, tasks and open items | Transcribe discussions word for word |
Four roles cover most incidents in small teams.
One rule from the SRE book is particularly helpful: only the technical lead and the people they bring in change the system during the incident. Anyone else with a “quick idea” suggests it but does not simply run it. That sounds bureaucratic, yet it prevents the classic situation where two people tweak the same configuration at the same time and afterwards nobody knows which change helped.
Combining roles in a small team
Four roles do not mean four people. As long as a role is unassigned, it stays with the incident commander. Google and Atlassian both describe this basic rule. From it follows a simple split by team size:
| People available | Split |
|---|---|
| 2 | Person A: incident commander + communication + scribe. Person B: technical lead. |
| 3 | A: incident commander + scribe. B: technical lead. C: communication. |
| 4 or more | Each role separately, additional people support the technical lead. |
With just one person there is nothing to coordinate. In that case another trick helps: bring in a second person early, even if only to handle communication. The one combination I would avoid is incident commander plus technical lead in a single person. That is exactly where you lose the overview, because your head is buried in the code.
With two or three people, the incident commander combines several roles.
When should you declare an incident?
The role model only helps if it actually kicks in. Many teams muddle along for too long because “declaring an incident” feels dramatic. The SRE book names three clear triggers, and I add a fourth:
- A second team is needed to solve the problem.
- Customers notice the error.
- The problem is still unsolved after an hour of focused analysis.
- You are unsure whether one of the three points applies.
I mean the fourth point seriously. An incident you close again after ten minutes costs almost nothing. One you only declare after three hours costs trust. Also, do not spend long debating severity. PagerDuty recommends assuming the highest plausible level when in doubt and correcting the classification in the postmortem. That saves time people would otherwise spend arguing about definitions.
Incident commander tasks step by step
Our example is a made-up bike rental: since 9:40 a.m., rentals in the app have been failing and the support inbox is filling up. Here is how I would proceed as incident commander.
1. Declare the incident and take the role
A dedicated channel (for example #inc-2026-11-05-rentals) and one sentence everyone can see: “I am incident commander for this incident. Rentals in the app have been failing since 9:40.” From now on, everything important goes through this channel. Add a video call once more than two people are actively working on it.
2. Assign roles by name
“Mia, you take technical lead. Jonas, you write the updates. I will keep the notes.” The name matters. A question to the room (“Can someone take a look at the database?”) almost always means that nobody feels addressed, or three people at once. PagerDuty rightly calls this the bystander effect.
3. Size up the situation
Before anyone fixes anything, collect the basics: what are the symptoms? Who is affected? Is it getting worse, staying the same or fluctuating? What changed recently? The result goes at the very top of the channel or into a shared document. Google calls this the “live incident state document” and describes keeping it current as the incident commander’s most important task.
4. Stabilize and decide
The technical lead proposes actions, the incident commander asks about risks and decides. Stopping the damage always comes first: roll back, switch off a feature, throttle load. The actual root cause can wait. One idea from PagerDuty has stuck with me: a wrong decision is better than no decision. A wrong decision can be corrected, standstill cannot.
Instead of “Does everyone agree?” you ask: “Are there any strong objections?” Then wait briefly. If nothing comes, go ahead. Asking for agreement creates endless rounds, while asking for strong objections surfaces only what really matters.
5. Assign tasks with a name and a time box
Every task gets a person and a time window: “Mia, roll back to yesterday’s version. Let me know in ten minutes whether the error rate drops.” The person confirms briefly. After ten minutes you check in, even if they have not reported back. That way no task gets lost in limbo.
6. Updates at a fixed cadence
Above all, stakeholders want to know when they will hear something next. Therefore communication sends updates at a fixed rhythm, for instance every 30 minutes, even when there is nothing new. “No change, we are testing hypothesis B, next update 10:45” is a perfectly good update. It keeps managers from coming into the incident channel to ask.
7. Verify, close, hand over
Once an action is done, you check with the technical lead whether it works. If not, go back to sizing up. If it does, declare the end of the incident as clearly as the start: “The incident is resolved, rentals have been working again since 10:52. I am closing the incident. Jonas owns the postmortem.” Without this sentence, people stay on the call out of uncertainty.
These four activities repeat until the incident is resolved. PagerDuty sums them up as Size-Up, Stabilize, Update, Verify. The animated diagram shows the cycle.
🎬 The incident commander loop – this animated diagram only runs on my site: watch it on sascha-ms.de
What an incident commander does not do
The list is short but hard to stick to. An incident commander does not fix things personally, does not read logs and does not click through dashboards. As soon as you do, the incident loses the person who holds the overview. This feels wrong especially to experienced developers, because they might find the bug faster. Maybe so. But while you are filtering in the log explorer, communication is waiting for your go-ahead and two other people are working on the same hypothesis.
If you notice that you really want to jump in yourself, that is a good signal for a handover: someone else takes command, and you switch to technical work.
Communication during an incident
Good incident communication is boring. Short sentences, a fixed structure, no speculation. In addition, it pays to acknowledge what you receive (“Got it, rollback is running”) so the other person knows their information arrived.
Status update template
You can copy this template straight into your wiki. It works for chat, email and status pages:
[Incident] Rentals in the app – update 2 – 10:15
Status: Cause suspected, action in progress
Impact: New rentals fail, ongoing rentals are not affected
Since: 09:40
What we do: Roll back to yesterday’s version
What you can do: Answer customer requests with the “rental incident” text snippet
Next update: 10:45 at the latest
Contact: Jonas (communication), channel #inc-2026-11-05-rentals
Phrases that help on the incident call
| Situation | Phrase |
|---|---|
| Chaos on the call | “One at a time. Mia first, then Ali.” |
| Decision | “We are rolling back. Any strong objections?” |
| Unclear answer | “I need a yes or no: is the rollback done?” |
| Curious manager | “The next update is at 10:45 in the channel. Do you want to take command?” |
| Debate about principles | “Good point for the postmortem. I will note it.” |
The question to the manager is adapted from the PagerDuty documentation and works surprisingly well. Either the person officially takes over, which leads to a clean handover. Or they decline and realize on their own that they are getting in the way.
Handing over command
Incidents sometimes last longer than a working day, and nobody coordinates well after five hours. A handover is therefore a perfectly normal part of the process. Four steps are enough for a clean handover:
- Brief the new person outside the call (chat message or short phone call).
- Walk through the current picture, open tasks, decisions made and the next update together.
- Announce the handover in the channel: “Ali is incident commander from now on.”
- Only leave once the new person has explicitly confirmed.
One detail from the PagerDuty documentation matters to me: command does not change automatically just because a more experienced person joins. Handovers happen on purpose, not because someone with a bigger title enters the call.
How to practice the incident commander role
Nobody becomes a good incident commander during their first real outage. The role needs practice beforehand, and that takes little effort.
Prepare a role card and a channel
Create a page in your wiki: the four roles, the triggers, the update template, the handover steps. Add a fixed naming scheme for incident channels. When things are on fire, nobody goes looking for processes. Then the first click has to land.
A small practice incident in the sprint
One hour per quarter is enough to start. One person makes up a scenario (“order confirmations are sent twice”), another takes the incident commander role, and everyone else plays along. It is not about the technical solution but about the process: who declares, who writes updates, how does the handover go? Finish with ten minutes of reflection, ideally in the style of a retrospective. PagerDuty calls such exercises Failure Friday, elsewhere they are known as game days. I show how to turn this into a full exercise with a playbook, a hypothesis and stop conditions in my guide on how to run a game day.
Rotate the incident commander role
The SRE book explicitly recommends rotating people through the roles. I would use this especially for junior developers. The role trains skills that rarely get practiced day to day: speaking clearly under pressure, setting priorities, delegating. Someone who first served as scribe and then once acted as incident commander with an experienced partner will be much calmer next time.
Common mistakes and how to avoid them
The team lead is always incident commander
This happens almost by itself, because everyone looks to the team lead and the incident commander role sticks to them. Yet it makes the team dependent on one person, who may well be on vacation. It also mixes role and hierarchy, which makes open disagreement on the call harder.
The incident commander joins the fixing
See above. When you notice it, hand over command. That is exactly what the handover is for.
Radio silence towards everyone else
The team works with focus, but nobody informs support or the business side. The result is phone calls, direct messages and people on the call who just want to know what is going on. A fixed update interval reliably solves this.
The incident never officially ends
The bug is gone, everyone quietly drops off the call, and the next day nobody knows who writes the postmortem. A clear end with a named owner for the follow-up prevents this.
My take
I consider the incident commander role one of the most underrated tools for small teams. Many people associate incident management with large corporations, on-call rotations and status pages. Yet small teams benefit the most, because every person counts there and duplicated work hurts immediately.
Where I disagree with the classic literature: the PagerDuty documentation expects an IC to have been involved in at least two major incidents beforehand. For small teams that is too strict. Their next real incident is often months away. Better to practice regularly and then simply do it. A nervous incident commander with a clear role card beats having none at all.
And a word to managers: often the best support during an incident is not joining the call. Read the updates, keep distractions away from the team and save your questions for the postmortem. They help more there.
📄 Printable cheat sheet – Roles, triggers, the IC loop, a status update template and a handover checklist on one page. Download PDF · View online
Who takes on the incident commander role in an emergency often depends on your on-call planning. I describe how to set that up fairly in the guide On-call rotation: how to organize it fairly.
FAQ
What does an incident commander do?
An incident commander coordinates the response to an incident. They assign roles and tasks, make decisions, keep the current picture up to date and ensure regular updates. Others handle the technical repair.
Does an incident commander need deep technical knowledge?
No. A rough understanding of the architecture is enough. Clear communication, decisiveness and the ability to listen to experts and adjust the plan matter more.
Who should be incident commander in a small team?
Anyone who has practiced the role and is not working on the bug themselves at the moment. Ideally the role rotates, so the team does not depend on the team lead or a single person.
Is an incident commander the same as an incident manager?
In most companies, yes. Atlassian uses “incident manager”, Google and PagerDuty use “incident commander”. Some companies, however, use incident manager for a permanent position that owns the whole process, beyond individual incidents.
Your next step
Print the cheat sheet, create a role card in your wiki and schedule a one-hour practice incident within the next four weeks. Let the person who is least often in the spotlight be incident commander. And I am curious: who coordinates in your team today when things are on fire, and was that a deliberate choice or did it just happen?
Sources
- Google SRE Book: Managing Incidents
- PagerDuty Incident Response: Incident Commander Training
- PagerDuty Incident Response: Scribe Training
- Atlassian: Incident response roles and responsibilities
This post was created with AI assistance.
👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (0)