Complete firstRunbook
At least one runbook exists for the planned failure mode so the game day can test instructions.
Without: Without a runbook, the game day becomes improvisation rather than validation, and learning rates stay low.
RunsheetDevOps
Game Day
ComplexityHigh
TimeHalber Tag
Participants5-20
FormatWorkshop
MaturityEstablished
01
Prerequisite
What needs to be finished first
Complete firstIncident Command
Incident roles (commander, communicator, operator) are established so the exercise runs realistically.
Without: Without roles, game day turns into a discussion and response times and escalation are not measured.
02
Preparation
What needs to be ready before start
Materials
Failure-injection tools (Chaos Mesh, Gremlin, AWS Fault Injection Simulator); observability stack (Datadog, Grafana, logs); communication channel; scenario and expectation script; safety shutoff.
People / roles
A game day master (script author); incident commander; operator team; observer for each relevant service; stakeholder representative; safety officer with abort authority.
Pre-read
Architecture diagram; SLO/SLA of affected services; known weaknesses from past incidents; current runbooks; pre-test health check.
Time needed
4-8 h for a game day, plus 1-2 weeks preparation
Setup
Schedule date in calendar and inform all relevant stakeholders. Production mode: failure injection on staging or isolated tenant. Define safety abort conditions (big red button). Establish communication plan so real users are not startled.
03
Core question
The one question this method answers
How do system, tools, and team react to realistically injected failures, and which gaps in runbooks, monitoring, or roles become visible?
04
Flow
Marker: Phase
| Step | Duration | Action | Hint |
|---|---|---|---|
1Phase 1: Script and hypotheses | 3-5 days lead time | Define three to five scenarios (for example DB failover, pod crash, region outage). For each scenario define hypothesis, expected result, and success criteria (for example recovery in 5 min). | Write hypothesis explicitly. If no one states expected behavior before start, no surprise can be learned. |
2Phase 2: Setup and safety brief | 30 min | Verify observability stack. Assign roles. Clarify Big Red Button criteria (for example real customer impact above threshold X). Communicate to stakeholders. | If stakeholders are not informed, a real alert can trigger escalations outside the exercise. Clear labeling is required. |
3Phase 3: Injection and response | 2-4 h | Scenario by scenario: inject failure, team responds with runbook, observers log response times, decisions, and tool behavior. Game day master delays recovery when appropriate. | Maintain realism. When participants know it is an exercise, adrenaline drops. Still, avoid too long delays to prevent frustration. |
4Phase 4: Recovery and verification | 30-60 min | After each scenario, team performs controlled recovery. Observers check whether monitoring detected the failure and runbook worked. | Recovery is part of the exercise. If recovery fails, the runbook has a gap. When complete standby outage happens, safety officer triggers Big Red Button. |
5Phase 5: Debrief and follow-ups | 60-90 min | Run after-action review per scenario: plan vs reality, root causes, lessons, actions. Create tickets with owner and deadline for top-3 lessons. Plan runbook updates. | Lessons without follow-up waste the game day. Limit to three actions per game day, otherwise follow-through disappears. |
05
Artifact
What comes out at the end
Form
Game day report with scenarios, hypotheses, response metrics (time to detect, time to mitigate), observations, lessons learned, and action list with owner and deadline.
Versioning / ownership
One entry per game day with date, scenarios, and outcomes. Compare across game days (time-to-detect and time-to-mitigate trends). Link runbook versions before and after game day.
Tool alternatives
- Gremlin or Chaos Mesh for failure injection
- AWS Fault Injection Simulator, Azure Chaos Studio
- Custom scripts with Toxiproxy or iptables
- Confluence or Notion report with attachments (logs, screenshots)
- PagerDuty drill mode for realistic alerting
game-day-working-template.md
Compact working template for Game Day with context, input, output artifacts, and next step.
Game Day Working Template
Goal
A realistic exercise to practice incident response and recovery.
Context
When and for what do we use this method?
Input
Which data, observations, decisions, or materials are available?
Execution
Short notes along the runsheet.
Output artifacts
- Simulation Notes:
- Gaps List:
- Updated Runbooks:
Assumptions and open questions
- ...
Decision / Next step
Owner, date, and success signal.
06
Example output
Concrete filled scenario, fictional example
game-day-beispiel.md
Concrete filled scenario, fictional example
Game Day - Platform team Workspot, 15.05.2026
Participants: Game Day Master @lisa, Incident Commander @ben, Operator (3 engineers), Observer (2 SREs), Safety Officer @marcus.
Scenarios
- DB primary failover.
- CDN region outage (eu-central).
- Auth service 100% 5xx for 10 min.
Hypotheses
- Scenario 1: Recovery in <3 min via auto failover.
- Scenario 2: Failover to eu-west via DNS in <5 min.
- Scenario 3: Runbook RB-AUTH-01 leads to mitigation in <10 min.
Results
| Scenario | Time-to-detect | Time-to-mitigate | Hypothesis met |
|---|---|---|---|
| 1 DB failover | 0:42 | 2:18 | Yes |
| 2 CDN outage | 1:54 | 14:30 | No (DNS TTL 300 s, manual switch required) |
| 3 Auth 5xx | 0:28 | 4:50 | Yes, runbook worked |
Top lessons
- CDN DNS TTL is 300 s and blocks failover. Lower to 60s.
- Auth runbook worked reliably with cycle time within limit.
- Observer tool reported CDN outage only after 1:54 (polling interval 60 s).
Actions
- GD-01: Reduce CDN DNS TTL to 60 s. Owner: @marcus, by 22.05.
- GD-02: Reduce CDN health-check polling to 15 s. Owner: @lisa, by 29.05.
- GD-03: Make a quarterly drill standard at 2x per quarter. Owner: @ben.
07
Pitfalls
Recognize symptoms and steer against them
Trap
Game day in production without safeguards
Symptom
Real users are affected, complaints escalate, and trust is damaged.
What to do
Start in staging, then isolated production tenant, and only then full production. Use safety officer and big red button before every injection.
Trap
No hypotheses beforehand
Symptom
Team "just watches" without expected behavior, so surprises are missed.
What to do
Write hypotheses before start. State expectations in advance. Deviation from hypothesis is the primary learning signal.
Trap
Lessons not implemented
Symptom
Three game days produce the same lessons (for example slow monitoring), without change.
What to do
Cap three actions, with owner and deadline. Quarterly review expectation: close old actions before adding new ones. Otherwise stop.
Trap
Stakeholders not informed
Symptom
Real alert triggers and escalates to leadership while the team is practicing.
What to do
Define communication plan: inform all relevant stakeholders (support, sales, leadership) before. Mark alert channels (for example [DRILL]) clearly.
Trap
Too many scenarios
Symptom
Seven scenarios in 4 hours, no debrief, lessons blur together.
What to do
Maximum three scenarios per 4-hour game day. Debrief directly after each scenario. Use realistic cycle times.
Trap
No observers
Symptom
No one logs response times, so debrief depends on memory.
What to do
At least one observer per scenario with timer and notes template. Response metrics must be facts, not anecdotes.
08
Stop criteria
Done signals checkable in under a minute
No failure-injection tools available and scenarios cannot be realistically triggered.
Safety officer and big red button not established; risk to real users.
Incident roles are not clearly filled, so response remains improvisation.
Runbooks for planned scenarios do not exist and hypotheses cannot be tested.
Stakeholder communication is not agreed and escalation risk is not assessed.
Service has active problems at the time, making extra stress unacceptable.
Finished the runsheet?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.