To run a game day means breaking something on purpose: planned, announced and with a clear stop button. Then you watch how your systems and your people respond. That sounds like a toy exercise at first. In practice, it is one of the cheapest insurance policies a team can buy.
The topic keeps me busy because almost every team has a runbook, an emergency plan or an escalation list somewhere that nobody has ever used under pressure. And 3 a.m. is a bad moment to discover that the link in the runbook is dead, or that only one person knows where the switch for maintenance mode lives. In this guide you get a playbook to prepare, facilitate and review your first game day, even without a dedicated SRE department.
In short
A game day is an announced failure drill in which a team tests technology, processes and collaboration under realistic conditions. The biggest payoff comes from the review, when observations turn into concrete actions with owners. The most common pitfall is a scenario without a hypothesis and without a stop condition, because then the drill can quickly turn into a real incident.
What you will learn in this article
- what a game day is and how it differs from chaos engineering
- which roles you need and who decides to abort
- how to define the scenario, hypothesis and stop conditions
- a playbook and an agenda for a half-day exercise
- how the review leads to real improvements
What is a game day?
A game day is a planned exercise in which a team simulates or deliberately triggers a failure. The AWS Well-Architected Framework puts it briefly: “A game day simulates a failure or event to test systems, processes, and team responses.” So you test more than the technology. You also check whether alerts arrive, whether the right people can be reached and whether communication with the outside world works.
The name comes from sports: game day shows whether the training paid off. In operations it works the other way round. The game day is the training, so that the real emergency hurts less.
Game day, chaos engineering and tabletop exercise compared
These terms often get mixed up. Here is a short comparison:
| Format | What happens | Risk | Good for |
|---|---|---|---|
| Tabletop exercise | The facilitator describes a failure, and the team talks through its response at the table or in a call. Nothing actually breaks. | very low | getting started, new team members, communication flows |
| Game day | A failure is triggered in a real environment, and the team responds as it would in an emergency. Technology and people are tested together. | low to medium, depending on the environment | runbooks, alerts, roles, collaboration |
| Chaos engineering | Controlled experiments test a hypothesis about system behavior, often automated and repeated. | low to medium, with a limited blast radius | technical resilience, regressions after deployments |
In the SRE book, Google describes a tabletop variant called “Wheel of Misfortune”: a facilitator picks two people as on-call, announces a page and gradually provides context, for example what the dashboards are showing right now. That is an excellent first step if your team does not yet dare to touch real systems.
Why teams should run a game day
The strongest reason is simple: whatever is never practiced rarely works in an emergency the way it was written down. Among the typical anti-patterns, the Well-Architected Framework lists procedures that exist on paper but are never practiced, decision makers from the business side who stay out of the loop, and lessons that never flow back into recovery processes.
A game day also uncovers things that no code review will find. For example, the alert fires, but it lands in a channel that nobody follows. Or the rollback is automated, but a permission is missing that only two people have.
For me, a game day is therefore mainly a leadership topic. It only works if nobody feels exposed when something goes wrong. That is exactly what it is for. If your first game day ends with a question of blame, you will not get honest answers at the second one.
Roles on game day
Anyone who wants to run a game day needs clear roles; otherwise everyone ends up discussing the root cause at once and nobody keeps an eye on the clock. In a small team, one person can take on several roles. Just keep the facilitator and the responders separate.
| Role | Task | Tip |
|---|---|---|
| Facilitator (game master) | prepares the scenario, triggers the failure, provides context and holds the playbook | the only person who knows the scenario in advance |
| Responders | react as in a real on-call shift: acknowledge the alert, analyze, fix | ideally the people who also handle real incidents |
| Incident commander | coordinates the response, assigns tasks, gives status updates | a good chance to practice the role |
| Scribe | writes down observations with timestamps: when did the alert arrive, when was what decided? | records facts, not judgments |
| Safety officer | watches the stop conditions and may end the exercise at any time | must not be a responder at the same time |
| Stakeholders | for example the business side or support, check communication and business impact | bring them in for the status updates |
I described how the incident commander role works in detail in my guide to the incident commander role in small teams. A game day is the ideal place to try this role without real pressure.
Prepare a game day in six steps
Preparation decides whether it succeeds. Before you can run a game day, you need a goal, a scenario and a playbook. For a half-day exercise, plan roughly two to four hours of preparation, more the first time.
1. Define the goal and the guiding question
Write down a question the game day should answer. Good guiding questions are concrete and verifiable, for example: “Do we notice within five minutes when the payment service stops responding, and can we get back to normal operation with the runbook alone?” A weak guiding question would be: “Let’s see how stable we are.”
2. Choose a scenario
Good scenarios come from three sources: past incidents, new features that have never failed under load, and dependencies you are merely hoping will hold. Take a fictional example, a ticket shop for concerts:
- The external payment service suddenly responds with an eight-second delay.
- The Lambda function for order confirmations throws an error on every tenth invocation.
- An API key for sending emails has expired.
- The person who knows the deployment best is on vacation and cannot be reached.
The last item is not a typo. A game day may also simulate organizational failures. Especially in small teams, knowledge that lives in a single head is often a bigger risk than any database.
3. Define the steady state and the hypothesis
Before you break anything, describe normal operation, the so-called steady state. This is a measurable value, such as the error rate, the 99th percentile latency or the number of successful orders per minute. The principles of chaos engineering recommend exactly this: define a measurable normal state and then make an assumption about how it behaves under disruption.
The hypothesis for the ticket shop could read: “If the payment service takes eight seconds, the circuit breaker kicks in after 30 seconds at most, customers see an understandable message, and the alert reaches the on-call person within five minutes.” Now you have something that can be clearly confirmed or refuted at the end.
4. Limit the blast radius and set stop conditions
The blast radius is the area a failure can hit at most. Limit it deliberately: a staging environment, a single endpoint, a small share of requests or a fixed time window. Also define hard stop conditions that end the exercise immediately, for example:
- The error rate in production rises above a defined threshold.
- A real alert outside the scenario fires.
- A customer contacts support with a matching problem.
- The safety officer says “stop”.
The way back matters, too. Write down how the failure is ended, and do a dry run of that path once beforehand. With AWS Fault Injection Service, you can tie stop conditions directly to CloudWatch alarms. I show how this works for Lambda functions in my article on chaos experiments with AWS FIS and Lambda.
5. Announce it and invite the right people
A game day is not a surprise party. Inform everyone who might be affected: your own team, neighboring teams, support and, if production is involved, the business side. The responders do not need to know which scenario is coming, but they should know that there will be an exercise that morning. Also plan to include someone from the business side, because many incidents turn out to be communication problems.
6. Write the playbook
The playbook is a short document that only the facilitator knows in full. You can copy and adapt this template:
GAME DAY – PLAYBOOK
Date / time: Thu, 10:00–12:30
Environment: staging (ticket shop), no production traffic
Guiding question: Can we detect and fix a slow payment service
with the existing runbook?
Steady state: error rate < 1 %, p99 checkout latency < 2 s
Hypothesis: circuit breaker kicks in after <= 30 s, alert after <= 5 min,
customers see a notice instead of an error page
Failure: raise the payment mock's response time to 8 s
Stop conditions: real alert outside the scenario, errors in production,
"stop" from the safety officer
Way back: reset the mock configuration, check the deployment
Roles: facilitator, responders (2), incident commander,
scribe, safety officer, support
Extra prompts: after 15 min: "Support reports 20 tickets about duplicate
charges" (only if the team is too fast)
Review: 11:45–12:30, same room / call
The extra prompts are a trick borrowed from tabletop exercises: if the team finds the failure too quickly or gets stuck, the facilitator can feed in new information and steer the scenario.
Game day agenda
If you want to run the game day smoothly, a fixed agenda helps. With this plan, half a day works well. For a pure tabletop exercise, 30 to 60 minutes is often enough.
| Time | Agenda item | Who |
|---|---|---|
| 10:00 | Briefing: goal, rules, stop conditions, roles | facilitator |
| 10:15 | Trigger the failure | facilitator |
| 10:15–11:15 | Response: detect, analyze, stabilize | responders, incident commander |
| ongoing | Record observations with timestamps | scribe |
| 11:15 | End the failure, check normal operation | facilitator, safety officer |
| 11:30 | Short break | everyone |
| 11:45–12:30 | Review | everyone |
🎬 How a game day runs – this animated diagram only runs on my site: watch it on sascha-ms.de
A briefing with clear rules
The briefing takes less than fifteen minutes, but it matters. Explain that the exercise is about the system and the processes, not about individuals. Read the stop conditions out loud. And agree on a code word so that everyone involved can tell a real incident from the drill, for example a prefix such as “DRILL” in every message in the incident channel.
Trigger the failure and observe
The facilitator triggers the failure, either by changing configuration, with a tool like AWS FIS or, in a tabletop exercise, by announcing an alert. After that, the facilitator holds back. The responders work the way they would in an emergency: with the real dashboards, the real incident channel and the real runbook.
The scribe is worth their weight in gold during this phase. Only facts with a time get written down, for example “10:21 alert in channel, 10:26 runbook opened, dashboard link broken”. Judgments belong in the review.
Step in or let it run?
The facilitator may step in if the team is stuck in a dead end and learning time is running out. A good prompt is new information, not a hint at the solution. Instead of “Take a look at the timeout configuration”, try: “The dashboard now shows that calls to the payment service take eight seconds on average.”
How the review leads to real improvements
The review is the part that justifies the whole effort. Schedule it right after the exercise, while everyone’s impressions are fresh. The format matches a blameless postmortem, as I described in my guide to the blameless postmortem: you look at processes and conditions, not at culprits.
Questions for the review
- Was the hypothesis confirmed or refuted? How can we tell?
- When did who notice the failure, and how?
- What in the runbook helped, what was missing, what was wrong?
- Where did we wait longest, and what for?
- Who should have been informed and was not?
- What would we do differently during a real outage at night?
Actions with an owner and a date
Every finding that requires a change goes into the backlog as a task, with an owner and a target date. That sounds trivial, yet it is where most game days fail. The Well-Architected Framework explicitly asks for this: document findings and track improvements to completion. At the next game day, the first thing you check is what happened to those actions.
How to run your first game day
No team needs to practice in production on day one. Anyone who wants to run a game day for the first time should start small. I recommend three stages:
- Tabletop exercise: 45 minutes in a meeting room or a call. The facilitator narrates the failure, and the team talks through its response. No risk, plenty of insight into communication and runbooks.
- Staging with a real failure: the failure is actually triggered in a test environment. Now you see whether alerts, dashboards and rollbacks really work.
- Production with a limited blast radius: only once the first two stages run safely, in a small, clearly bounded slice and with automatic stop conditions.
There is no fixed rule for the cadence. The Well-Architected Framework talks about regular, scheduled game days. I consider once a quarter a good starting point: often enough to keep the knowledge fresh, and rarely enough to implement the actions in between. If you are currently organizing an on-call rotation, a short tabletop exercise is also a good way to onboard new people.
Common game day mistakes
No hypothesis
Without a hypothesis, you end up with impressions instead of a result. Write down what you expect beforehand, and measure it.
Nobody owns the abort
If nobody is explicitly responsible for aborting, nobody dares to call it. Name a person who is not working on the fix themselves.
Testing only the technology
Many teams test nothing but failover and retries. Yet real incidents often fail on communication: who informs support, who decides on a maintenance page? So bring in at least one person from outside the development team.
Looking for someone to blame
As soon as sentences like “Why didn’t you …” come up in the review, the learning culture is at risk. Redirect such questions: “What would have helped you in that moment?”
Results fizzle out
A game day without implemented actions is expensive theater. Plan the tasks into the next sprint and make their status visible at the next game day.
My take
I consider game days one of the most underrated tools for small teams. Running a game day costs half a day. That is much cheaper than the first real incident in which it turns out the emergency plan never worked. At the same time, game days are a gauge of team culture: if a team talks openly about gaps after a game day, that is a good sign. If everyone stays silent, you should take a closer look as a leader.
Where I disagree: game days are sometimes staged as a chaos engineering show, with a big stage and spectacular outages. No team needs that. A boring tabletop exercise that results in three concrete improvements is worth more than an exciting production outage that nobody talks about honestly afterwards.
FAQ
How often should you run a game day?
Regularly and on a schedule. Once a quarter is a good starting point. After major architecture changes or new dependencies, an extra session is worthwhile.
What is the difference between a game day and chaos engineering?
Chaos engineering uses controlled, often automated experiments to check the technical behavior of a system. A game day additionally exercises how people respond: alerting, roles, communication and runbooks.
Do you need a production environment for a game day?
No. The best way to start is a tabletop exercise or a staging environment. Production only comes into question once stop conditions and the way back have been tested.
How long does a game day take?
A tabletop exercise takes 30 to 60 minutes. A game day with a real failure and a review fits well into half a day.
Who should take part in a game day?
The people who also handle real incidents, plus a facilitator, a scribe and at least one person from support or the business side for communication.
Your next step
Want to run a game day yourself soon? Take your team’s last real incident and turn it into a 45-minute tabletop exercise. All you need is the playbook above, a calendar slot and someone to act as facilitator. I am curious: which failure would you practice first, the technical one or the one where your most important person is on vacation?
Sources
- AWS Well-Architected Framework: REL12-BP05 Conduct game days regularly
- AWS Well-Architected: game day definition
- Google SRE Book: Accelerating SREs to On-Call and Beyond (Disaster Role Playing)
- Principles of Chaos Engineering
This post was created with AI assistance.
👉 Next step: Not sure which AWS service fits your use case? My AWS service finder asks a few questions and points you to the right one.
📬 Stay updated: New articles on serverless, AI and leading tech teams via RSS, browser push or the weekly newsletter (in German).
Originally published on sascha-ms.de. More articles on serverless, AI in everyday development and leading tech teams: sascha-ms.de/en
Top comments (0)