What needs to be finished first
A recurring operational activity or defined incident type is identified (at least 2-3 occurrences in 90 days or clearly foreseeable risk).
Roles and escalation paths in incident are known so runbook can reference concrete responsibilities.
What needs to be ready before start
Repository for runbooks (markdown in Git or wiki); template with header (Trigger, Symptoms, Severity, Owner), prerequisites, steps, verification, rollback, escalation; links to dashboards, logs and tools.
One runbook owner (usually SRE or Service Owner); one reviewer from operational team; one test pilot who executes runbook under drill conditions.
Existing tickets, postmortems or records showing the flow during real occurrence; logs, metrics dashboards, involved services; known special cases.
Creation 2-6 h per runbook, test 1-2 h, review annually or after change
Create template. Naming convention (service + trigger). Unified storage location. Direct link from alerts to runbook URL. Visible in incident channel.
The one question this method answers
Which steps does a trained person execute in which order to handle the trigger safely without having to improvise?
Marker: Sektion
| Step | Duration | Action | Hint |
|---|---|---|---|
1Section 1: Header and trigger | 20 min | Fill header: title, service, owner, reviewer, last update, severity. Describe trigger and symptoms concretely (alert name, metric threshold, log entries). | If symptoms are vague, runbook is pulled on wrong alert. Concrete example log entry or screenshot helps enormously. |
2Section 2: Prerequisites and access | 20 min | Tools, permissions, VPN, configurations needed for execution. List pre-checks: what must be present before step 1 starts. | Nothing is more frustrating than reaching step 4 and not having tool access. Pre-checks force permissions to be maintained upfront. |
3Section 3: Steps | 60-120 min | Per step: number, action (command or UI instruction), expected response, next step. Decision points as If/Then. Check idempotency per step. | Steps must be concrete enough that someone without prior experience can execute them. No implicit assumptions. Better too detailed than too short. |
4Section 4: Verification and rollback | 20 min | Verification steps: how operator knows action succeeded. Rollback steps if effect does not occur or side effect appears. | Verification is not "see if it runs". Concrete command or metric with threshold. Rollback is mandatory section, not optional. |
5Section 5: Escalation and postcheck | 15 min | When to escalate, to whom, with what context. Postcheck: what follows afterward (close ticket, start postmortem, inform owner). | Escalation trigger explicit: after 30 min without effect, on severity increase, on security indicators. Otherwise operator hesitates too long in incident. |
6Section 6: Drill and review | 1-2 h per drill | Test pilot executes runbook on staging or in Game Day. Incorporate feedback. Set review cadence (for example semiannual or after change). | A runbook that was never practiced will not work in incident. At least one drill after creation and after larger change. |
What comes out at the end
Markdown or wiki entry with named sections, clear naming convention, links to logs, dashboards and tools, plus change log at end.
Git-versioned with pull request review. Last update date visible. Changes with date and reason in footer. Annual Review as mandatory calendar appointment.
Checklist for operational runbooks with trigger, diagnosis, action, rollback, and escalation.
Concrete filled scenario, fictional example
Concrete filled scenario, fictional example
Service: order-api. Owner: SRE Team. Severity: Sev-2. Last updated: 2026-05-12.
Trigger: Alert order-api-5xx-rate > 1.5% over 5 min.
Symptoms: Datadog dashboard order-api, tile "5xx Rate" red. Logs contain TimeoutException more often than 10/min.
Pre-checks
Steps
kubectl -n order get pods -> all pods Ready? If no, step 3.order-api-db: connection pool exhausted? If yes, step 4. Otherwise step 5.kubectl -n order rollout restart deployment order-api. Wait 2 min. Verification step 6.kubectl -n order scale deployment order-api --replicas=8 (standard is 4). Wait 3 min. Verification step 6.kubectl -n order logs -l app=order-api --tail=200 | grep ERROR. Identify pattern. If unknown, escalate.Rollback: After stabilization from step 4, set replicas back to 4.
Escalation: After 30 min without effect to Service Owner @ben (+49-...) and SRE Lead @lisa. Raise severity to Sev-1.
Postcheck: Close PagerDuty incident, create postmortem in Confluence, ticket for Root Cause Analysis in JIRA.
Change log: 2026-05-12 - Added step 4 after incident INC-2026-031. Owner: @lisa.
Recognize symptoms and steer against them
In incident, command has wrong syntax or tool is not installed.
Drill after creation mandatory. At least annual Game Day with runbook. Feed drill errors back into runbook.
Tool commands no longer fit, service has different name, thresholds outdated.
Annual Review as appointment. Review runbook on every larger service change. Last update visible in header, old runbooks (>1 year) marked.
Step 3 expects prior knowledge, operator stalls.
During creation use junior test: can someone without prior experience execute the step? If no, split step. Link glossary.
Step worsens situation, nobody knows how to revert.
Rollback is mandatory section. At least one reversal option per destructive step. If none possible, mark explicitly ("Attention: irreversible").
Operator struggles 2 h before getting help.
Concrete trigger: after X minutes without success, escalate to Y with Z context. Escalation is not failure, but part of protocol.
Operator does not know runbook exists or cannot find it.
Alert configuration contains runbook URL. Naming convention allows search by service. Incident bot posts link on page.
Done signals checkable in under a minute
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.