Complete firstIncident Timeline Analysis
A chronological reconstruction of the event with timestamps, signals, decisions and reactions is available.
Without: Without timeline, hypotheses rely on memory; cause work becomes speculation.
RunsheetOperations
Root Cause Analysis
ComplexityMedium
Time1-4 h
Participants3-8
FormatWorkshop + async
MaturityCanonical
01
Prerequisite
What needs to be finished first
Complete firstBlameless Postmortem
A culture and format of blameless review are established so systemic causes instead of people are addressed.
Without: Without this norm, RCA slips into blame assignment, data is withheld, causes remain hidden.
02
Preparation
What needs to be ready before start
Materials
Whiteboard or digital board with RCA template (Problem, Symptoms, Hypotheses, Data, Causes, Actions); logs, metrics, configuration states; postmortem template; method tools such as 5 Whys, Fishbone, Fault Tree for selection.
People / roles
One facilitator with postmortem experience; 4-8 people with direct system contact from development, ops, support; one scribe; for larger incidents additionally one neutral investigator.
Pre-read
Incident timeline; relevant logs and metrics; available postmortems of similar incidents; symptom description as 1-2 sentence statement; affected customers or systems.
Time needed
2-4 h for initial RCA, then follow-up sessions depending on complexity
Setup
Symptom at top of board as one sentence. List available methods (5 Whys for linear causality, Fishbone for broad hypothesis branching, Fault Tree for safety-critical systems). Postmortem template with sections Problem, Impact, Timeline, Root Cause, Countermeasures, Lessons Learned.
03
Core question
The one question this method answers
Which systemic causes caused or enabled the problem, and which countermeasures prevent recurrence instead of only treating symptoms?
04
Flow
Marker: Phase
| Step | Duration | Action | Hint |
|---|---|---|---|
1Phase 1: Specify problem | 20-30 min | Describe symptom concretely: what, when, where, how often, which effect. Quantify impact (users, revenue, data integrity). Delimit against similar problems. | If problem statement already contains causes ("because X did not work"), that belongs in hypothesis phase. Clean symptom is basis for unbiased search. |
2Phase 2: Hypothesis fan | 30-45 min | Brainstorm possible causes across relevant categories (for example code, configuration, infrastructure, data, process, human). Fishbone or mind map as structure. Separate assumptions and evidenced causes. | If only two or three hypotheses appear, search is too narrow. Force at least one hypothesis per category, even if unlikely. Avoid confirmation bias. |
3Phase 3: Test hypotheses | 45-90 min | Find data per hypothesis: logs, metrics, config diffs, witnesses. Verify or falsify. Remaining plausible hypotheses are candidates for Root Cause. | If no data source can be named per hypothesis, it is speculative. Hypothesis test is core of method. "Sounds plausible" is no verification. |
4Phase 4: Set Root Cause(s) | 30-45 min | Identify Root Cause(s) from verified hypotheses. Often multiple causes plus triggering factor. For every cause check: controllable, systemic or individual case. | A single Root Cause is suspicious. Complex systems usually have multiple interacting causes. Common endpoints "human error" are not Root Cause, but indicator of missing safety net. |
5Phase 5: Countermeasures and Lessons Learned | 30-45 min | Countermeasure per Root Cause with owner and date. Separate symptom fix (hotfix), recurrence prevention (structural), detection (monitoring). Document Lessons Learned. | If all countermeasures are monitoring or training, structural change is missing. Per Root Cause at least one action that prevents recurrence, not only improves visibility. |
05
Artifact
What comes out at the end
Form
Postmortem document with sections Problem, Impact, Timeline, Hypotheses, checked data, Root Cause(s), countermeasures with owner and date, Lessons Learned. Reference logs, metrics and optional supplementary methods (5 Whys, Fishbone) as appendix.
Versioning / ownership
Date, incident ID, author and decider in header. Append later findings as update section at end, do not overwrite. If Root Cause later proves false, status to `revised`, keep original analysis.
Tool alternatives
- Markdown in repo under docs/postmortems/<date>-<incident-id>.md
- Confluence page in postmortem space
- Notion database with filter by service and date
- Linear or Jira issue with linked postmortem
root-cause-analysis-working-template.md
Compact working template for Root Cause Analysis with context, input, output artifacts, and next step.
Root Cause Analysis Working Template
Goal
Structured analysis to find the effective causes of a problem behind its symptoms.
Context
When and for what do we use this method?
Input
Which data, observations, decisions, or materials are available?
Execution
Short notes along the runsheet.
Output artifacts
- Problem Statement:
- Cause Hypotheses:
- Confirmed Causes:
- Action Plan:
Assumptions and open questions
- ...
Decision / Next step
Owner, date, and success signal.
06
Example output
Concrete filled scenario, fictional example
root-cause-analysis-beispiel.md
Concrete filled scenario, fictional example
RCA - Incident 2026-05-14-001 Order API 500 Spike
Symptom: On 2026-05-14 09:12-09:47 UTC, Order API returned HTTP 500 for 8% of checkout requests. Impact: 1,247 failed orders, estimated revenue loss 18,200 EUR.
Checked hypotheses
- Hypothesis fan with Fishbone: 11 hypotheses, categorized by code, config, infrastructure, data, process.
- Verified: connection-pool exhaustion (master DB), triggered by batch job without own pool quota.
- Falsified: hypothesis "hardware replica defect" (replica showed symptoms only after stabilization, not causal).
Root Causes
- Structural: batch and online traffic share same connection pool without quota separation.
- Process: code review for batch workloads contained no pool check, reviewer missed implication.
- Detection: pool exhaustion not alerted, only 500 rate.
Countermeasures
- Dedicated connection pool for batch (structural). Owner @anna, by 2026-05-21.
- Checklist for batch jobs in CODEOWNERS review (process). Owner @ben, by 2026-05-28.
- Pool saturation alert in Datadog (detection). Owner @sabine, by 2026-05-23.
Lessons Learned
Shared-resource architecture without quota separation is a class of risks; similar patterns exist in worker threads and cache slots, audit scheduled for 2026-06-30.
07
Pitfalls
Recognize symptoms and steer against them
Trap
Personification as Root Cause
Symptom
RCA ends at "engineer did not check", no systemic cause, no structural countermeasure.
What to do
If person appears as cause, ask further: which safety net was missing. Person is symptom, missing checks or tools are Root Cause.
Trap
Single Root Cause
Symptom
Complex incident reduced to one cause, other contributing factors disappear.
What to do
List factors explicitly: triggering, contributing, enabling. In complex systems, usually 3-5 factors are involved, not one.
Trap
Hypotheses without data
Symptom
Hypotheses accepted or rejected by plausibility, no logs or metrics consulted.
What to do
Name at least one data source per hypothesis. If none available, mark hypothesis as unverified and plan data collection spike, do not speculate.
Trap
Countermeasure addresses symptom
Symptom
Actions are monitoring, alert or hotfix, no structural change.
What to do
Mandatory per Root Cause: one structural countermeasure (architecture, process, tool). Detection separate. Hotfix only short-term stabilization with follow-up.
Trap
Confirmation bias
Symptom
First plausible hypothesis becomes truth, others checked superficially.
What to do
Assign devil's advocate or second person for parallel hypothesis fan. Document falsification attempts explicitly, not only verifications.
Trap
No follow-up tracking
Symptom
Countermeasures documented but not implemented, same incident repeats months later.
What to do
Countermeasures as issues in ticket system with owner and date. Check status every 2 weeks. Repeat incident triggers RCA review, not new RCA.
08
Stop criteria
Done signals checkable in under a minute
Symptom is not reproducible and data missing, causal chain cannot be checked.
People with direct system contact unavailable, hypotheses are hearsay.
Blameless norm missing, culture allows only blame assignment, RCA becomes culpability finding.
Time window under 2 h and no follow-up meetings, method cannot be played through.
Political pressure toward predetermined result, RCA becomes theater.
Incident was one-off without structural risk; simplified logging and Lessons Learned suffice, full RCA oversized.
Finished the runsheet?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.