Complete firstClear scope
One targeted, randomizable change with reliable measurement and enough traffic for the planned horizon.
Without a boundary, data, perspectives, or flows become mixed.
Method guideGrowth
A/B Testing
A practical method guide: clarify prerequisites and materials, understand the complete process, and look up outputs, pitfalls, and stop criteria.
ComplexityHigh
Time1-4 Wochen
Participants1-6
FormatAsync
MaturityCanonical
01
Prerequisite
What needs to be finished first
Complete firstAccess to evidence
Relevant people and sources can be included.
Gaps remain visible; missing evidence must not be invented.
02
Preparation
What needs to be ready before start
Materials
Shared workspace, accessible sources, and a decision log.
People / roles
Experiment owner · Engineering · Analytics · reviewer
Pre-read
Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
Time needed
Prework for brief and instrumentation; run to the pre-set experiment horizon.
Setup
Before randomization, set unit, allocation, primary metric, guardrails, MDE, sample, and horizon. Preselect one analysis approach: a classical fixed-horizon analysis read at the end, or an appropriate sequential method. Do not change the rule after viewing interim results.
03
Core question
The one question this method answers
Does B produce the pre-defined meaningful effect over A without breaching guardrails?
04
Flow
Phases with goal, actions, output and pitfall
A/B testing compares variants under controlled allocation and connects effect, uncertainty, and guardrails to a pre-defined decision.
Variant comparison and pre-defined decision
Fictional worked example
Compare variants under controlled allocation and make a pre-defined decision using effect and guardrails.
Primary metric
Specify the primary outcome and its denominator before assignment.
Bookings per randomly assigned person.
Design
Record allocation, planned sample, calendar horizon, and interference controls.
50/50 random assignment; 20,000 people per arm, at least 14 days; no concurrent checkout changes.
Measures
Enter the estimated effect and uncertainty, plus each guardrail.
Fictional: booking A 11.8%, B 12.3%, difference +0.5 percentage points; 95% interval −0.2 to +1.2. Error rate 0.8% for both.
Decision
Apply the decision rule chosen before the test; record uncertainty.
No clear advantage at the pre-set horizon; result is inconclusive. Do not extend because of interim results.
Fictional read at the pre-set horizon: 20,000 people per arm, at least 14 days. A 11.8%, B 12.3%, difference +0.5 percentage points, 95% interval −0.2 to +1.2; guardrail error rate 0.8% each. No clear advantage; inconclusive.
- 1
Decision
Predefine the decision
Goal
Is analysis fixed before seeing results?
Inputs for this step
- Hypothesis, baseline, MDE, guardrails
Actions
- Write the hypothesis, primary metric, guardrails, MDE, and ship/hold/stop rule.
- Is analysis fixed before seeing results?
Output
Pre-registered experiment brief
Pre-registered experiment brief
Quality check
Is analysis fixed before seeing results?
Pitfall
Primary metric changed after launch Version the metric and decision rule before randomization.
Timing: Part of the agreed work block; research may be separate.
- 2
Design
Design a valid test
Goal
Will groups remain comparable?
Inputs for this step
- Randomization unit, traffic, and interference map
Actions
- Set randomization unit, 50/50 allocation, required sample, and minimum duration using baseline and MDE.
- Will groups remain comparable?
Output
Randomized design with sample and horizon
Randomized design with sample and horizon
Quality check
Will groups remain comparable?
Pitfall
Stopping early on a favorable fluctuation Analyze at the planned horizon or preselect a valid sequential method.
Timing: Part of the agreed work block; research may be separate.
- 3
Launch
Instrument and launch
Goal
Does exposure record actual assignment?
Inputs for this step
- Feature flag, event QA, and rollback
Actions
- QA assignment and events; launch only after tracking and rollback checks.
- Does exposure record actual assignment?
Output
QA-checked rollout and exposure data
QA-checked rollout and exposure data
Quality check
Does exposure record actual assignment?
Pitfall
Allocation is not random or stable Check assignment and sample-ratio mismatch; pause if broken.
Timing: Part of the agreed work block; research may be separate.
- 4
Analysis
Decide at the horizon
Goal
Was a fixed horizon used without peeking?
Inputs for this step
- Pre-set sample, horizon, and analysis rule
Actions
- Analyze at the set horizon or under the preselected sequential rule; inspect effect interval and guardrails.
- Was a fixed horizon used without peeking?
Output
Decision with effect interval and guardrail review
Decision with effect interval and guardrail review
Quality check
Was a fixed horizon used without peeking?
Pitfall
Guardrail worsens despite primary gain Honor the pre-set guardrail; do not ship based on significance alone.
Timing: Part of the agreed work block; research may be separate.
05
Artifact
What comes out at the end
Sketch: what the artifact looks like
Form
Versioned result with question, evidence, open issues, next action, ownership, and review date.
Versioning / ownership
Record date, scope, sources, and changes. Preserve earlier observations.
Tool alternatives
- Whiteboard or paper
- Shared spreadsheet
- Versioned project space
ab-testing-working-template.md
Prepare A/B Testing with a clear question, roles, and sources.
A/B Testing: Worksheet
Question
To be confirmed
Desired outcome
To be confirmed
Scope
Plan the full method Time: Set according to scope and available evidence
Complete this worksheet on paper or in your own document during the work.
Preparation for this scope
Method setup
- Materials: Shared workspace, accessible sources, and a decision log.
- Roles: Experiment owner · Engineering · Analytics · reviewer
- Advance information: Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
- Overall time needed: Prework for brief and instrumentation; run to the pre-set experiment horizon.
- Setup: Before randomization, set unit, allocation, primary metric, guardrails, MDE, sample, and horizon. Preselect one analysis approach: a classical fixed-horizon analysis read at the end, or an appropriate sequential method. Do not change the rule after viewing interim results.
Prepare for the work steps
Decision: Predefine the decision
- Hypothesis, baseline, MDE, guardrails
Design: Design a valid test
- Randomization unit, traffic, and interference map
Launch: Instrument and launch
- Feature flag, event QA, and rollback
Analysis: Decide at the horizon
- Pre-set sample, horizon, and analysis rule
Variant comparison and pre-defined decision
Primary metric
Specify the primary outcome and its denominator before assignment.
...
Design
Record allocation, planned sample, calendar horizon, and interference controls.
...
Measures
Enter the estimated effect and uncertainty, plus each guardrail.
...
Decision
Apply the decision rule chosen before the test; record uncertainty.
...
Blank worksheet: start with a bounded question. Enter observed facts only, mark assumptions, and record open questions and sources.
Frequentist Sequential Testing · Statsig Docs
Decision: Predefine the decision
Expected artifact: Pre-registered experiment brief
Pre-registered experiment brief
Entry:
...
- Is analysis fixed before seeing results?
Design: Design a valid test
Expected artifact: Randomized design with sample and horizon
Randomized design with sample and horizon
Entry:
...
- Will groups remain comparable?
Launch: Instrument and launch
Expected artifact: QA-checked rollout and exposure data
QA-checked rollout and exposure data
Entry:
...
- Does exposure record actual assignment?
Analysis: Decide at the horizon
Expected artifact: Decision with effect interval and guardrail review
Decision with effect interval and guardrail review
Entry:
...
- Was a fixed horizon used without peeking?
Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/ab-testing/run-sheet
06
Example output
Concrete filled scenario, fictional example
Fictional example for orientation. Counts and timings describe this case and are not universal requirements.
ab-testing-beispiel.md
Concrete filled scenario, fictional example
Fictional read at the pre-set horizon: 20,000 people per arm, at least 14 days. A 11.8%, B 12.3%, difference +0.5 percentage points, 95% interval −0.2 to +1.2; guardrail error rate 0.8% each. No clear advantage; inconclusive.
07
Pitfalls
Recognize symptoms and steer against them
Quick overview, the full text is in the matching phase in the flow
- Predefine the decisionPrimary metric changed after launchView in phase
- Design a valid testStopping early on a favorable fluctuationView in phase
- Instrument and launchAllocation is not random or stableView in phase
- Decide at the horizonGuardrail worsens despite primary gainView in phase
Trap
Predefine the decision
Symptom
Primary metric changed after launch
What to do
Version the metric and decision rule before randomization.
Trap
Design a valid test
Symptom
Stopping early on a favorable fluctuation
What to do
Analyze at the planned horizon or preselect a valid sequential method.
Trap
Instrument and launch
Symptom
Allocation is not random or stable
What to do
Check assignment and sample-ratio mismatch; pause if broken.
Trap
Decide at the horizon
Symptom
Guardrail worsens despite primary gain
What to do
Honor the pre-set guardrail; do not ship based on significance alone.
08
Stop criteria
When to pause, resolve prerequisites, or choose another approach
The question or boundary remains ambiguous: narrow it first.
Key data or participants are missing: label the gap and arrange access.
The group mixes observation and explanation: revisit the evidence.
An immediate safety or operational risk appears: stop and use the responsible procedures.
Finished the method guide?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.