Move from a simulated outage through practiced incident response to a documented, improved learning loop.
A practiced incident flow with clarified roles, working communication, and documented improvements to the deployment flow.
You can decide which improvements to the deployment flow are prioritized for implementation.
Carry the derived improvements into runbooks and schedule another game day.
Can the system and response team safely detect and handle this bounded failure scenario?
A simulated outage uncovers weaknesses before they cause costs in a real incident.
The weaknesses surfaced in the game day show whether incident response is workable.
Who commands, who works the technical problem, and how do we maintain a shared situation?
A clear role structure prevents time being lost to ownership questions during a real incident.
The clarified roles need a shared communication channel to work in a real emergency.
Which operational action benefits from shared visibility, and what controls prevent misuse?
A bundled channel makes the incident timeline traceable for everyone involved and speeds up coordination.
After the exercise, the whole flow is reviewed in a structured way to learn from it.
What was meant to happen, what actually happened, what can we learn, and what should we change or sustain?
The structured review ensures the insights from the exercise actually turn into improvements.
The derived improvements are carried into runbooks and the deployment process.
Artifacts stay collapsed until you actually need them.
Prepare Game Day with a clear question, roles, and sources.
# Game Day: Worksheet
## Question
To be confirmed
## Desired outcome
To be confirmed
## Scope
Plan the full method
Time: Set according to scope and available evidence
Complete this worksheet on paper or in your own document during the work.
## Preparation for this scope
### Method setup
- **Materials:** Shared workspace, accessible sources, and a decision log.
- **Roles:** Exercise lead · safety lead · response team · observers
- **Advance information:** Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
- **Overall time needed:** Recurring exercise; plan preparation, execution, recovery, and action review separately.
- **Setup:** Create a shared version. Label facts, assumptions, and decisions separately.
### Prepare for the work steps
#### Objective: Choose objective and risk
- Resilience goal, service criticality, and risk
#### Boundaries: Prepare participants and stop controls
- Approved scenario, participants, and kill switch
#### Execution: Run the scenario under control
- Monitoring, observer log, and user-impact limit
#### Learning: Track learning
- Findings, remediations, and next exercise date
## Safety-bounded scenario and exercise learning
### Scenario
State learning goal, service, scenario, and environment.
...
### Safety boundary
Define participants, observation, kill switch, and thresholds.
...
### Expected signals
Capture alerts, system state, and response during exercise.
...
### Learning
Record recovery, findings, owners, and next exercise.
...
Blank worksheet: start with a bounded question. Enter observed facts only, mark assumptions, and record open questions and sources.
[Conduct game days regularly · AWS Well-Architected](https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/rel_testing_resiliency_game_days_resiliency.html)
## Objective: Choose objective and risk
Expected artifact: Bounded, approved failure scenario
Bounded, approved failure scenario
Entry:
...
- [ ] Is the scenario approved and bounded?
## Boundaries: Prepare participants and stop controls
Expected artifact: Participant, communications, and safety plan
Participant, communications, and safety plan
Entry:
...
- [ ] Can anyone stop the exercise?
## Execution: Run the scenario under control
Expected artifact: Observed system and team response
Observed system and team response
Entry:
...
- [ ] Is real user protection maintained?
## Learning: Track learning
Expected artifact: Tracked findings and retest plan
Tracked findings and retest plan
Entry:
...
- [ ] Do gaps feed procedures and the next exercise?
## Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/game-day/run-sheet
Prepare Incident Command with a clear question, roles, and sources.
# Incident Command: Worksheet
## Question
To be confirmed
## Desired outcome
To be confirmed
## Scope
Plan the full method
Time: Set according to scope and available evidence
Complete this worksheet on paper or in your own document during the work.
## Preparation for this scope
### Method setup
- **Materials:** Shared workspace, accessible sources, and a decision log.
- **Roles:** Incident Commander · Operations · Communications · scribe
- **Advance information:** Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
- **Overall time needed:** For the duration of the incident; roles, update cadence, and handoffs adapt to the situation.
- **Setup:** Create a shared version. Label facts, assumptions, and decisions separately.
### Prepare for the work steps
#### Command: Declare and lead
- Incident symptoms, severity, and responders
#### Situation: Establish situation
- Confirmed facts, impact, and systems
#### Coordination: Coordinate work and communication
- Roles, workstreams, and update audience
#### Handover: Stabilize and hand off
- Stability evidence, open risks, and handoff
## Situation, workstreams, and handoff
### Situation
Record severity, Incident Commander, and start time.
...
### Roles
Separate confirmed facts, hypotheses, impact, and open questions.
...
### Cadence
Assign workstreams, update cadence, and audience.
...
### Handoff
Document stability criteria, residual risk, and accepted handoff.
...
Blank worksheet: start with a bounded question. Enter observed facts only, mark assumptions, and record open questions and sources.
[Incident management at Google · Google Cloud](https://cloud.google.com/blog/products/gcp/incident-management-at-google-adventures-in-sre-land)
## Command: Declare and lead
Expected artifact: Incident declaration and commander
Incident declaration and commander
Entry:
...
- [ ] Is one person clearly accountable for coordination?
## Situation: Establish situation
Expected artifact: Shared situation and impact summary
Shared situation and impact summary
Entry:
...
- [ ] Are unresolved assumptions marked?
## Coordination: Coordinate work and communication
Expected artifact: Coordinated technical and communications work
Coordinated technical and communications work
Entry:
...
- [ ] Can responders help without conflicting parallel commands?
## Handover: Stabilize and hand off
Expected artifact: Documented stabilization and command handoff
Documented stabilization and command handoff
Entry:
...
- [ ] Are service and command responsibilities clearly handed off?
## Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/incident-command/run-sheet
Prepare ChatOps with a clear question, roles, and sources.
# ChatOps: Worksheet
## Question
To be confirmed
## Desired outcome
To be confirmed
## Scope
Plan the full method
Time: Set according to scope and available evidence
Complete this worksheet on paper or in your own document during the work.
## Preparation for this scope
### Method setup
- **Materials:** Shared workspace, accessible sources, and a decision log.
- **Roles:** Service owner · Security · bot owner · users
- **Advance information:** Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
- **Overall time needed:** Several steps from use-case selection and permissions through pilot; then review access and use regularly.
- **Setup:** Create a shared version. Label facts, assumptions, and decisions separately.
### Prepare for the work steps
#### Operation: Choose a suitable operation
- Recurring operation and user need
#### Controls: Constrain commands and permissions
- Permissions, allowlist, and approval policy
#### Feedback: Bring feedback into chat
- Test environment, bot integration, and error paths
#### Review: Review use and risks
- Audit records, use, and security findings
## Controlled operational command with audit
### Command
Describe operation, user need, and chat channel.
...
### Controls
Record command permission, identity, approval, and rollback.
...
### Feedback
Add test result, error cases, and chat feedback.
...
### Audit
Record audit source, owner, and next security review.
...
Blank worksheet: start with a bounded question. Enter observed facts only, mark assumptions, and record open questions and sources.
[Using ChatOps to help Actions on-call engineers · GitHub Blog](https://github.blog/engineering/infrastructure/using-chatops-to-help-actions-on-call-engineers/)
## Operation: Choose a suitable operation
Expected artifact: Suitable, bounded ChatOps use case
Suitable, bounded ChatOps use case
Entry:
...
- [ ] Is chat the right place for this action?
## Controls: Constrain commands and permissions
Expected artifact: Least-privilege command and approval policy
Least-privilege command and approval policy
Entry:
...
- [ ] Are irreversible actions protected?
## Feedback: Bring feedback into chat
Expected artifact: Tested command with useful feedback
Tested command with useful feedback
Entry:
...
- [ ] Are errors actionable and safe?
## Review: Review use and risks
Expected artifact: Auditable trace and review path
Auditable trace and review path
Entry:
...
- [ ] Can every action be attributed to an identity?
## Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/chatops/run-sheet
Plan a protected review of a completed event with an intent, timeline, and follow-up.
# After-Action Review: Worksheet
## Question
To be confirmed
## Desired outcome
To be confirmed
## Scope
Review one event together
Time: 20–45 minutes after the event; investigation and implementation are additional
Complete this worksheet on paper or in your own document during the work.
## Preparation for this scope
### Method setup
- **Materials:** Goal or plan, timeline data and relevant work records; a safe conversation space, facilitator, and shared notes.
- **Roles:** Facilitation keeps the questions clear and invites participation. Participants reconstruct events. A recorder captures lessons, actions, owners, and a later review.
- **Advance information:** Gather the goal, plan, timeline, decisions, and observable facts. Mark uncertainty explicitly.
- **Overall time needed:** 20–45 minutes for a bounded review. Deep investigation, formal incident reports, and implementation take additional work.
- **Setup:** Start with four questions: What was supposed to happen? What actually happened? What worked or hindered the task, and why? What will we change next time? Compare the plan and event before explaining causes. Ask open questions, separate facts from hypotheses, and finish with a few owned actions. Army doctrine is the historical reference; adapt the process and language to civilian contexts.
### Prepare for the work steps
#### Intent: Clarify the goal and boundary
- Plan, goal, context
#### Sequence: Reconstruct actual events
- Timestamps, records, participant perspectives
#### Compare: Contrast intent and reality
- Shared intent and timeline
#### Learn: Check conditions and lessons
- Comparison, causal hypotheses, evidence
#### Adapt: Set actions and a later check
- Lessons, owners, follow-up event
## Review record in five sections
### Goal and plan
What should happen, by when, and with what expected result? Bound the period under review.
...
### Actual sequence
Which observable events, decisions, and dates are supported? Mark what is unknown.
...
### Comparison and conditions
What differed from the plan? Which helpful or difficult conditions are supported by sources?
...
### Lessons and hypotheses
What should we sustain or change? Which explanation is evidenced, and which still needs a check?
...
### Actions and later check
Which few changes have an owner, date, and observable signal in a relevant future event?
...
Blank worksheet: keep the prompts and record your own facts, observations, and actions. The fictional example is not copied over.
[U.S. Army, NTC EXOP Annex A: After Action Review Standards, adapted for civilian work](https://home.army.mil/irwin/application/files/1816/9455/6714/FY23_JUNE_2023_NTC_EXSOP_RELEASEABLE.pdf)
## Intent: Clarify the goal and boundary
Expected artifact: Shared starting point
Without a clear expectation, differences are hard to interpret.
Entry:
...
- [ ] The review opens with general opinions about the event.
## Sequence: Reconstruct actual events
Expected artifact: Evidence-based timeline
A plausible explanation is not yet a verified event.
Entry:
...
- [ ] Memories conflict and are judged immediately.
## Compare: Contrast intent and reality
Expected artifact: Comparison with differences
A difference is a learning prompt, not personal blame.
Entry:
...
- [ ] Discussion jumps from a difference to an accusation.
## Learn: Check conditions and lessons
Expected artifact: Lessons and open questions
A single explanation can miss systemic or situational conditions.
Entry:
...
- [ ] The first explanation is accepted as the cause.
## Adapt: Set actions and a later check
Expected artifact: Improvement plan
An action without a later check has no measured effect.
Entry:
...
- [ ] The action list grows too long or lacks owners.
## Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/after-action-review/run-sheet