Purpose
When two variants compete, discussions quickly decide by taste rather than effect. A/B Testing checks behavior under controlled conditions and separates real improvement from chance or expectation effects.
Growth Research method
A/B Testing
A/B testing compares variants under controlled allocation and connects effect, uncertainty, and guardrails to a pre-defined decision.
A/B testing compares variants under controlled allocation and connects effect, uncertainty, and guardrails to a pre-defined decision.
Does B produce the pre-defined meaningful effect over A without breaching guardrails?
- Participants
- 1-6
- Time
- 1-4 Wochen
Phases may be revisited; their order supports the work but does not replace domain judgment.
Decision
Predefine the decision
Write the hypothesis, primary metric, guardrails, MDE, and ship/hold/stop rule.
Design
Design a valid test
Set randomization unit, 50/50 allocation, required sample, and minimum duration using baseline and MDE.
Launch
Instrument and launch
QA assignment and events; launch only after tracking and rollback checks.
Analysis
Decide at the horizon
Analyze at the set horizon or under the preselected sequential rule; inspect effect interval and guardrails.
Review the result and open assumptions at the next suitable checkpoint. Record changes, ownership, and new evidence.
A good fit when
Use when: One targeted, randomizable change with reliable measurement and enough traffic for the planned horizon.
Ideal for
- Conversion optimization
- Messaging tests
- Feature validation
Not good for
- Very small traffic
- Unclear hypotheses
Outputs
Working picture
The artifact records the hypothesis, primary metric, guardrails, MDE, and ship/hold/stop rule.
Testable evidence
QA assignment and events; launch only after tracking and rollback checks.
Next step
The analysis reports effect and uncertainty at the planned horizon, accounts for guardrails, and documents the decision.
Variant comparison and pre-defined decision
Fictional worked example
Compare variants under controlled allocation and make a pre-defined decision using effect and guardrails.
Primary metric
Specify the primary outcome and its denominator before assignment.
Bookings per randomly assigned person.
Design
Record allocation, planned sample, calendar horizon, and interference controls.
50/50 random assignment; 20,000 people per arm, at least 14 days; no concurrent checkout changes.
Measures
Enter the estimated effect and uncertainty, plus each guardrail.
Fictional: booking A 11.8%, B 12.3%, difference +0.5 percentage points; 95% interval −0.2 to +1.2. Error rate 0.8% for both.
Decision
Apply the decision rule chosen before the test; record uncertainty.
No clear advantage at the pre-set horizon; result is inconclusive. Do not extend because of interim results.
Fictional read at the pre-set horizon: 20,000 people per arm, at least 14 days. A 11.8%, B 12.3%, difference +0.5 percentage points, 95% interval −0.2 to +1.2; guardrail error rate 0.8% each. No clear advantage; inconclusive.
First step
Start with a bounded question, clarify access and participants, and collect available evidence. Compare variants under controlled allocation and make a pre-defined decision using effect and guardrails.
Good to do first
Deep dive
How it works
A change is tested in isolation against a reference variant while a clear success measure runs in parallel. Only the observed difference in behavior decides whether the difference is robust enough to change product or communication.
In detail
A/B Testing grounds product or communication decisions in observed behavior. Quality comes from a clear hypothesis, clean randomization, sufficient runtime, and a decision threshold fixed in advance. As a pure numbers exercise the test stays shallow; it becomes solid when result, counter-metrics, and learning value are interpreted together.
Facilitation
Before starting, hypothesis, primary metric, guardrails, and stopping rules must be fixed in writing, or the analysis tips into wishful thinking. The decisive point is reached when the variant stays cleanly isolated and no one adjusts metrics along the way to fit. Close with a product decision and the learning points, even if the effect doesn't show up.
The method guide walks execution through 4 phases, timing, 4 pitfalls, and clear stop criteria.
Open method guideYour perspective
Does this method help with your work?
Your feedback helps shape the next improvements to methodatlas.
Sources and topics
Artifact templates
A/B Testing: WorksheetPrepare A/B Testing with a clear question, roles, and sources.markdown
A/B Testing: WorksheetPrepare A/B Testing with a clear question, roles, and sources.
markdownab-testing-working-template.md
Prepare A/B Testing with a clear question, roles, and sources.
A/B Testing: Worksheet
Question
To be confirmed
Desired outcome
To be confirmed
Scope
Plan the full method Time: Set according to scope and available evidence
Complete this worksheet on paper or in your own document during the work.
Preparation for this scope
Method setup
- Materials: Shared workspace, accessible sources, and a decision log.
- Roles: Experiment owner · Engineering · Analytics · reviewer
- Advance information: Clarify the question, timeframe, data access, roles, and known uncertainties in advance.
- Overall time needed: Prework for brief and instrumentation; run to the pre-set experiment horizon.
- Setup: Before randomization, set unit, allocation, primary metric, guardrails, MDE, sample, and horizon. Preselect one analysis approach: a classical fixed-horizon analysis read at the end, or an appropriate sequential method. Do not change the rule after viewing interim results.
Prepare for the work steps
Decision: Predefine the decision
- Hypothesis, baseline, MDE, guardrails
Design: Design a valid test
- Randomization unit, traffic, and interference map
Launch: Instrument and launch
- Feature flag, event QA, and rollback
Analysis: Decide at the horizon
- Pre-set sample, horizon, and analysis rule
Variant comparison and pre-defined decision
Primary metric
Specify the primary outcome and its denominator before assignment.
...
Design
Record allocation, planned sample, calendar horizon, and interference controls.
...
Measures
Enter the estimated effect and uncertainty, plus each guardrail.
...
Decision
Apply the decision rule chosen before the test; record uncertainty.
...
Blank worksheet: start with a bounded question. Enter observed facts only, mark assumptions, and record open questions and sources.
Frequentist Sequential Testing · Statsig Docs
Decision: Predefine the decision
Expected artifact: Pre-registered experiment brief
Pre-registered experiment brief
Entry:
...
- Is analysis fixed before seeing results?
Design: Design a valid test
Expected artifact: Randomized design with sample and horizon
Randomized design with sample and horizon
Entry:
...
- Will groups remain comparable?
Launch: Instrument and launch
Expected artifact: QA-checked rollout and exposure data
QA-checked rollout and exposure data
Entry:
...
- Does exposure record actual assignment?
Analysis: Decide at the horizon
Expected artifact: Decision with effect interval and guardrail review
Decision with effect interval and guardrail review
Entry:
...
- Was a fixed horizon used without peeking?
Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/ab-testing/run-sheet
Alternatives and neighbors
Fitting alternatives with decision guidance, followed by other related methods.
UX Research
Usability Testing
Instead of A/B Testing, when you want to see direct usage behavior instead of discussing assumptions within the team.
Growth
Pirate Metrics AARRR
Maps the AARRR funnel, sets baselines, and surfaces leaks that translate directly into experiments.
Growth
Flywheel
Maps a recurring growth loop, marks friction points, and derives reinforcing levers.
Product Discovery
Smoke Test
Tests demand with minimal effort, reads conversion signals, and turns them into a clear learning note.
Product Strategy
Counter Metrics
Moves target group, value, goals, and priorities toward a concrete result through "name the main metric", "collect possible unintended side effects", and "use counter metrics alongside the main metric in reviews".
Product Discovery
Experiment Canvas
Moves customer problem, solution idea, and evidence toward a concrete result through "clarify the learning goal and the assumption to test", "formulate the hypothesis as an if-then statement", and "define the decision rule after the test".
Constraints and origin
HighAsync1-4 Wochen
Profile
- Method form
- Research method
- Participants
- 1-6
- Time
- 1-4 Wochen
- Maturity
- Canonical
- Origin
- Praxisgemeinschaft
Next up: Growth Experiment