Complete firstBounded subject
The scope and purpose of Counter Metrics are stated as a concrete question.
Without scope, counter-metric contract becomes arbitrary.
Method guideProduct Strategy
Counter Metrics
A practical method guide: clarify prerequisites and materials, understand the complete process, and look up outputs, pitfalls, and stop criteria.
ComplexityLow
Time30-60 min
Participants2-6
FormatWorkshop + async
MaturityEstablished
01
Prerequisite
What needs to be finished first
Complete firstAccess to relevant perspectives
Primary metric and decision, Potential side effects, Segment and system data and domain decision makers are available.
Without these perspectives, central claims in counter-metric contract remain unsupported.
02
Preparation
What needs to be ready before start
Materials
Primary metric and decision, Potential side effects, Segment and system data, shared workspace, source links, and decision log.
People / roles
Product · Analytics · affected domain roles · Risk/Compliance
Pre-read
Collect Primary metric and decision, Potential side effects, Segment and system data in advance and mark open assumptions.
Time needed
45–60 minutes initially; review per experiment
Setup
Prepare one primary metric with explicit counter-hypotheses, guardrails, and stop thresholds visibly and calibrate assessment terms before starting.
03
Core question
The one question this method answers
Which material degradation could the target metric hide, and which counter metric stops or qualifies the decision?
04
Flow
Phases with goal, actions, output and pitfall
Counter metrics state what must not degrade while the primary metric improves. They come from counter-hypotheses, system boundaries, and affected groups, and need equally clear formulas and decision thresholds.
counter-metric contract
Fictional worked example
one primary metric with explicit counter-hypotheses, guardrails, and stop thresholds as a precise working structure with a fictional example.
| Item | Primary metric | Counter-hypothesis | Counter metric | Guardrail | Decision |
|---|---|---|---|---|---|
| counter-metric contractWorked example | Completed reviews / started reviews; target +10%. | Fast shallow completions increase. | Reviews <5 min / completed reviews per week. | Stop at +2 pp; new teams separately. | Roll out only if primary target passes without guardrail breach. |
1. Primary metric
- counter-metric contract
- Completed reviews / started reviews; target +10%.
2. Counter-hypothesis
- counter-metric contract
- Fast shallow completions increase.
3. Counter metric
- counter-metric contract
- Reviews <5 min / completed reviews per week.
4. Guardrail
- counter-metric contract
- Stop at +2 pp; new teams separately.
5. Decision
- counter-metric contract
- Roll out only if primary target passes without guardrail breach.
The example demonstrates the domain logic of Counter Metrics; worksheets remain blank.
Microsoft Experimentation Platform: Overall Evaluation Criterion
- 1
Counter-hypotheses
Counter-hypotheses
Goal
Produce harm hypotheses so the core question “Which material degradation could the target metric hide, and which counter metric stops or qualifies the decision?” can be answered with stronger support.
Inputs for this step
- Primary metric and decision
- Potential side effects
- Segment and system data
Actions
- State plausible incentives, displacement, and affected groups for each target metric.
- Record source, rationale, and open uncertainty directly in harm hypotheses.
Output
harm hypotheses
harm hypotheses · source · uncertainty · next decision
Quality check
Is the path from optimisation to harm traceable?
Pitfall
A local target improves while a material side effect remains unobserved. Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Timing: 10–20 minutes
- 2
Counter metrics
Counter metrics
Goal
Produce metric contracts so the core question “Which material degradation could the target metric hide, and which counter metric stops or qualifies the decision?” can be answered with stronger support.
Inputs for this step
- harm hypotheses
- Potential side effects
- Segment and system data
Actions
- Define a measurable counter metric with population, formula, and window for each harm hypothesis.
- Record source, rationale, and open uncertainty directly in metric contracts.
Output
metric contracts
metric contracts · source · uncertainty · next decision
Quality check
Does the counter metric represent harm rather than a convenient proxy?
Pitfall
A local target improves while a material side effect remains unobserved. Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Timing: 10–20 minutes
- 3
Set thresholds
Set thresholds
Goal
Produce guardrail rule so the core question “Which material degradation could the target metric hide, and which counter metric stops or qualifies the decision?” can be answered with stronger support.
Inputs for this step
- metric contracts
- Potential side effects
- Segment and system data
Actions
- Set baseline, stop threshold, segment rules, and decision logic before optimisation.
- Record source, rationale, and open uncertainty directly in guardrail rule.
Output
guardrail rule
guardrail rule · source · uncertainty · next decision
Quality check
Is it clear when a positive primary result is rejected?
Pitfall
A local target improves while a material side effect remains unobserved. Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Timing: 10–20 minutes
- 4
Review outcomes
Review outcomes
Goal
Produce balanced decision so the core question “Which material degradation could the target metric hide, and which counter metric stops or qualifies the decision?” can be answered with stronger support.
Inputs for this step
- guardrail rule
- Potential side effects
- Segment and system data
Actions
- Evaluate primary and counter metrics together and add newly discovered side effects.
- Record source, rationale, and open uncertainty directly in balanced decision.
Output
balanced decision
balanced decision · source · uncertainty · next decision
Quality check
Are segment and long-term effects included?
Pitfall
A local target improves while a material side effect remains unobserved. Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Timing: 10–20 minutes
05
Artifact
What comes out at the end
Sketch: what the artifact looks like
Form
Versioned counter-metric contract with date, sources, owners, and next review.
Versioning / ownership
Retain date, source, assumption state, and decision for every version of counter-metric contract.
Tool alternatives
- counter-metric contract as board or document
- Source and evidence register
- Decision and action log
counter-metrics-working-template.md
Plan Counter Metrics around its domain steps and the concrete decision question.
Counter Metrics: Worksheet
Question
To be confirmed
Desired outcome
To be confirmed
Scope
Establish and review the practice Time: 45–60 minutes initially; review per experiment
Complete this worksheet on paper or in your own document during the work.
Preparation for this scope
Method setup
- Materials: Primary metric and decision, Potential side effects, Segment and system data, shared workspace, source links, and decision log.
- Roles: Product · Analytics · affected domain roles · Risk/Compliance
- Advance information: Collect Primary metric and decision, Potential side effects, Segment and system data in advance and mark open assumptions.
- Overall time needed: 45–60 minutes initially; review per experiment
- Setup: Prepare one primary metric with explicit counter-hypotheses, guardrails, and stop thresholds visibly and calibrate assessment terms before starting.
Prepare for the work steps
Counter-hypotheses: Counter-hypotheses
- Primary metric and decision
- Potential side effects
- Segment and system data
Counter metrics: Counter metrics
- harm hypotheses
- Potential side effects
- Segment and system data
Set thresholds: Set thresholds
- metric contracts
- Potential side effects
- Segment and system data
Review outcomes: Review outcomes
- guardrail rule
- Potential side effects
- Segment and system data
counter-metric contract
| Item | Goal · formula · desired change | Potential harm and causal path | Formula · population · window | Baseline · stop threshold · segment | Primary + counter · action |
|---|---|---|---|---|---|
| counter-metric contract |
- counter-metric contract: Worked example
Enter only real input, sources, and decisions.
Microsoft Experimentation Platform: Overall Evaluation Criterion
Counter-hypotheses: Counter-hypotheses
Expected artifact: harm hypotheses
harm hypotheses · source · uncertainty · next decision
Entry:
...
- Is the path from optimisation to harm traceable?
Counter metrics: Counter metrics
Expected artifact: metric contracts
metric contracts · source · uncertainty · next decision
Entry:
...
- Does the counter metric represent harm rather than a convenient proxy?
Set thresholds: Set thresholds
Expected artifact: guardrail rule
guardrail rule · source · uncertainty · next decision
Entry:
...
- Is it clear when a positive primary result is rejected?
Review outcomes: Review outcomes
Expected artifact: balanced decision
balanced decision · source · uncertainty · next decision
Entry:
...
- Are segment and long-term effects included?
Open questions and next steps
...
Method guide: https://methodatlas.meierhoff-systems.de/en/methods/counter-metrics/run-sheet
06
Example output
Concrete filled scenario, fictional example
Fictional example for orientation. Counts and timings describe this case and are not universal requirements.
counter-metrics-beispiel.md
Concrete filled scenario, fictional example
Fictional example: primary metric +10% completed reviews. Counter metric 1: share of reviews under five minutes must not rise by >2 pp because shallow completion is a risk. Counter metric 2: revisions to saved decisions within 48 hours must not exceed 15%. Segment check: new teams separately. Any breach stops rollout despite a positive primary metric.
07
Pitfalls
Recognize symptoms and steer against them
Quick overview, the full text is in the matching phase in the flow
- Counter-hypothesesA local target improves while a material side effect remains unobserved.View in phase
- Counter metricsA local target improves while a material side effect remains unobserved.View in phase
- Set thresholdsA local target improves while a material side effect remains unobserved.View in phase
- Review outcomesA local target improves while a material side effect remains unobserved.View in phase
Trap
Counter-hypotheses
Symptom
A local target improves while a material side effect remains unobserved.
What to do
Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Trap
Counter metrics
Symptom
A local target improves while a material side effect remains unobserved.
What to do
Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Trap
Set thresholds
Symptom
A local target improves while a material side effect remains unobserved.
What to do
Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
Trap
Review outcomes
Symptom
A local target improves while a material side effect remains unobserved.
What to do
Trace the concrete harm path to an observable leading or outcome measure and set the threshold before launch.
08
Stop criteria
When to pause, resolve prerequisites, or choose another approach
counter-metric contract answers the core question with traceable evidence or explicitly marked uncertainty.
Owners, next actions, and review dates are named for open points.
Finished the method guide?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.