What needs to be finished first
A testable hypothesis exists in written form with expected effect, success metric, and plausible effect size.
The affected conversion step is identified in the funnel and the baseline rate is known.
At least one counter-metric is defined that makes harm from manipulation or side effects visible.
What needs to be ready before start
Experiment tool with bucketing and statistical analysis; experiment brief template (hypothesis, variants, metrics, sample, duration, stop rule); sample-size calculator.
An experiment owner (Product or Growth); engineer for implementation; data analyst for evaluation; reviewer for brief approval; stakeholder for go-live decision.
Baseline conversion and variance; known seasonal effects; minimum detectable effect (MDE) as the target; tracking plan for success and counter-metrics; tests already running to check for interference.
1-2 weeks setup, plus 1-4 weeks runtime
Write the experiment brief and have it approved by the reviewer. Calculate sample size with MDE, alpha (0.05), and power (0.8). Implement the variant behind a feature flag. Run a dry test with an internal group before rollout.
The one question this method answers
Does the variant deliver a statistically significant effect on the success metric without degrading the counter-metric?
Marker: Phase
| Step | Duration | Action | Hint |
|---|---|---|---|
1Phase 1: Brief and sample size | 2-4 h | Write the experiment brief: hypothesis (if-then-because), variants A/B (or more), success metric, counter-metrics, MDE, sample per variant, runtime, stop rules, decision rule. | Choose the MDE realistically. If the MDE is a 1% absolute conversion lift but the baseline is 5%, tens of thousands of users are needed. Document the sample calculation explicitly. |
2Phase 2: Implementation and QA | 1-5 days | Implement the variant behind a feature flag. Check random bucketing (50/50 split). Test tracking events for success and counter-metrics. Run QA in staging plus internal dogfooding. | Bucketing errors can ruin the experiment invisibly. Sample Ratio Mismatch (SRM) is the most common source of error. Run a pre-test health check with a chi-square test. |
3Phase 3: Rollout and monitoring | 1-4 weeks | Roll out gradually (1% -> 10% -> 50%) to test stability. Monitor crashes, latency, and complaints daily. Do not interpret the success metric before the sample is reached. | Peeking too early massively increases the false-positive rate. If a tool offers sequential tests, use them. Otherwise: do not look. |
4Phase 4: Analysis | 1-2 days | After the sample is reached, evaluate each variant: point estimate, confidence interval, p-value, practical effect size. Check counter-metrics. Run segment analysis for heterogeneity. | p < 0.05 alone is not enough. The effect size must be practically relevant and counter-metrics must not degrade significantly. Otherwise the decision rule must be explicit. |
5Phase 5: Decision and rollout | 1 day | Decision: Ship, Iterate, or Kill. If Ship: plan 100% rollout and clean up the test code. If Iterate: formulate the next hypothesis. If Kill: remove the variant and note the lessons learned. | Every experiment yields knowledge, including a Kill. Store lessons learned in the knowledge base, otherwise the next test will search for the same effect again. |
What comes out at the end
Experiment report with brief, sample-size rationale, final values (point, CI, p), counter-metrics, segment analysis, decision, and follow-up. Supplemented by tracking plan and brief change log.
Unique ID and folder per experiment with brief, code references, analysis, and decision. Version briefs and document changes during runtime. Keep the archive for completed tests searchable.
Compact working template for A/B Testing with context, input, output artifacts, and next step.
Compares two or more variants using defined success metrics.
When and for what do we use this method?
Which data, observations, decisions, or materials are available?
Short notes along the runsheet.
Owner, date, and success signal.
Concrete filled scenario, fictional example
Concrete filled scenario, fictional example
Hypothesis: If the booking CTA appears above the fold on mobile, the conversion from detail to booking start rises from 33% to at least 38%, because fewer users miss the button.
Variants: A (control, CTA below description), B (sticky top CTA).
Metrics
Sample: 12,800 users per variant (MDE 5 pp, power 0.8, alpha 0.05). Sample window 18 days.
Result (runtime 18 days, n=25,840)
| Metric | A | B | Delta | p |
|---|---|---|---|---|
| Conv-to-start | 33.1% | 39.4% | +6.3pp | <0.001 |
| Cancellation rate | 4.2% | 4.4% | +0.2pp | 0.31 |
| Support tickets | 12 | 14 | +2 | 0.62 |
Segment: iOS +7.1pp, Android +5.4pp. Both significant.
Decision: Ship. Roll out to 100% on 27.05. Follow-up hypothesis: test a sticky CTA on desktop for the tablet layout.
Recognize symptoms and steer against them
Variant A and B end up with clearly different user counts despite 50/50 allocation.
Abort the test and review the bucketing code. Common causes: the variant loads too slowly and users drop off, or the cookie logic excludes users. Run a chi-square test automatically for every experiment.
After 3 days the variant looks better, and the team ends the test early.
Use a fixed runtime or a sequential test before the start. Early stopping without correction creates false positives. Make the decision only after the sample is reached.
Several metrics are evaluated in parallel and at least one becomes significant by chance.
Name one primary success metric in advance. Mark secondary metrics as exploratory. Use Bonferroni or Benjamini-Hochberg correction when there are multiple primary goals.
The effect is significant, but only 0.1 pp; implementation effort outweighs the benefit.
Expand the decision rule: not only significance, but also a minimum effect size. If the practical threshold is not reached, do not ship.
Click rate rises, but activation or retention falls below variant B.
Choose counter-metrics intentionally broader, including at least one lagging metric (activation, retention). If they significantly worsen, do not ship even with a positive primary outcome.
Multiple experiments share users and the same success metric, and the results conflict.
Check for interference before the start: do the experiments affect the same metric or UI area? If there is conflict, split them orthogonally (different layers) or run them serially.
Done signals checkable in under a minute
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.