Method guideProduct Discovery

Wizard of Oz Test

A practical method guide: clarify prerequisites and materials, understand the complete process, and look up outputs, pitfalls, and stop criteria.

ComplexityHigh

Time2-10 Tage

Participants5-20 Nutzer

FormatWorkshop + async

MaturityEstablished

Plan

Prerequisite

What needs to be finished first

Complete firstSimulability

An operator can produce intended responses within realistic bounds.

Unrealistic perfection would misrepresent the eventual product.

Complete firstParticipant protection

Plan voluntary research participation, data access, withdrawal and prompt debriefing.

If the concealed role is unjustifiable, test openly manual or transparently simulated service.

Preparation

What needs to be ready before start

Materials

Facade, operator console, response rules, log.

People / roles

Participants, moderator, concealed operator.

Pre-read

Define an interaction hypothesis, simulated scope and acceptable boundaries.

Time needed

60–90 minutes preparation; 20–40 minutes per session plus debriefing.

Setup

Prepare the user interface and operator console separately; display response rules and use shared timestamps in both logs.

Core question

The one question this method answers

How do people use this interaction when the missing technology is simulated under controlled conditions?

Flow

Phases with goal, actions, output and pitfall

In a Wizard of Oz Test a person behind the scenes operates functions that appear automated to participants. This tests interaction and value assumptions before building the technology.

Wizard of Oz Test· worksheet

Fictional example

In a Wizard of Oz Test a person behind the scenes operates functions that appear automated to participants. This tests interaction and value assumptions before building the technology.

Hypothesis and facade

Which interaction and assumption are tested? What is manually simulated?

Fictional: a course assistant suggests classes matching three stated time windows. Hypothesis: people can then choose a suitable option independently. A concealed operator selects from a fixed course list; no real AI exists.

Operator rules and limits

Define response rules, latency, failure cases and manual interventions.

Lea returns at most three matching offers within 20–40 seconds. No match triggers a request for another time window. Unplanned responses are logged as deviations.

Participation and debriefing

What information is needed upfront, which simulation is temporarily concealed, and how is debriefing handled?

Voluntary participation in a research prototype; only fictional scheduling data, human access to inputs disclosed. The exact operator workflow is explained immediately afterwards; withdrawal and agreed deletion are available.

Observation and operator log

Record task, user action, operator action, timing, help and outcome separately.

P02 enters three times; Lea responds after 32 seconds. P02 overlooks location and chooses a time-compatible class at the wrong venue. The moderator explains only after task completion.

Debriefing and learning decision

What was disclosed, how did the person respond and what limits remain?

P02 learns that Lea selected the options. The team adds location comparison and retests. Automated quality, scaling and operating costs are not established by this manual trial.

All example details are fictional and illustrate the application.

NN/g: The Wizard of Oz Method in UX

1
Simulation
Simulation
Goal
Simulation contract
Inputs for this step
- Interaction concept
Actions
- Define hypothesis, visible function and manually simulated scope.
- Set realistic response quality and latency; do not treat human skill as evidence of technical capability.
Output
Simulation contract
Is it clear which assumption the test can answer?
Quality check
Is it clear which assumption the test can answer?
Pitfall
The test is expected to prove scalability too. Plan technical feasibility as separate evidence.
Timing: 15–20 minutes
2
Protocol
Protocol
Goal
Execution protocol
Inputs for this step
- Simulation contract and tasks
Actions
- Prepare operator rules, boundaries, consent and debriefing.
- Minimise concealed information; do not hide human access to sensitive inputs or simulate consequential risky decisions.
Output
Execution protocol
Are data access, withdrawal and debriefing clear?
Quality check
Are data access, withdrawal and debriefing clear?
Pitfall
Unexpected sensitive data reaches the operator. Interrupt and use only agreed harmless test data.
Timing: 20–30 minutes
3
Rehearsal
Rehearsal
Goal
Rehearsed setup
Inputs for this step
- Protocol and facade
Actions
- Rehearse the facade and operator workflow with normal, failure and boundary cases.
- Verify moderator–operator coordination and complete time and action logs.
Output
Rehearsed setup
Is operation reproducible rather than improvised?
Quality check
Is operation reproducible rather than improvised?
Pitfall
Each participant receives a different special solution. Define response rules and flag deviations.
Timing: 20–30 minutes
4
Observation
Observation
Goal
Behaviour log
Inputs for this step
- Rehearsed setup
Actions
- Give neutral tasks; log user action, operator action and delay separately.
- Observe failures in expectation, trust or flow without moderator rescue; allow voluntary withdrawal throughout.
Output
Behaviour log
Is assistance visible as an intervention in the log?
Quality check
Is assistance visible as an intervention in the log?
Pitfall
Success after help is counted as independent use. Count assisted and unassisted completion separately.
Timing: 20–40 minutes
5
Debriefing
Debriefing
Goal
Test findings
Inputs for this step
- Logs and conversation
Actions
- Explain the manual simulation, collect reactions and bound the findings.
- Honour agreed options for handling research data; assess value, usability and unresolved automation separately.
Output
Test findings
Have the simulation and its limits been disclosed?
Quality check
Have the simulation and its limits been disclosed?
Pitfall
Results are presented as working automation. State the human contribution explicitly and define the next technical test.
Timing: 10–15 minutes

Artifact

What comes out at the end

Sketch: what the artifact looks like

Wizard of Oz · Human behind the CurtainTest a seemingly finished experience while the team manually performs critical system parts behind the scenes

Form

Interaction findings with operator actions, latency, boundaries and debriefing.

Versioning / ownership

Record date and scope; retain the previous version and reason when changing it.

Tool alternatives

Interactive prototype
Concealed operator console
Timestamped action log

wizard-of-oz-test-working-template.md

Define an interaction hypothesis, simulated scope and acceptable boundaries.

Wizard of Oz Test: Worksheet

Question

To be confirmed

Desired outcome

To be confirmed

Scope

Complete cycle Time: 60–90 minutes preparation; 20–40 minutes per session plus debriefing.

Complete this worksheet on paper or in your own document during the work.

Preparation for this scope

Method setup

Materials: Facade, operator console, response rules, log.
Roles: Participants, moderator, concealed operator.
Advance information: Define an interaction hypothesis, simulated scope and acceptable boundaries.
Overall time needed: 60–90 minutes preparation; 20–40 minutes per session plus debriefing.
Setup: Prepare the user interface and operator console separately; display response rules and use shared timestamps in both logs.

Prepare for the work steps

Simulation: Simulation

Interaction concept

Protocol: Protocol

Simulation contract and tasks

Rehearsal: Rehearsal

Protocol and facade

Observation: Observation

Rehearsed setup

Debriefing: Debriefing

Logs and conversation

Wizard of Oz Test· worksheet

Hypothesis and facade

Which interaction and assumption are tested? What is manually simulated?

...

Operator rules and limits

Define response rules, latency, failure cases and manual interventions.

...

Participation and debriefing

What information is needed upfront, which simulation is temporarily concealed, and how is debriefing handled?

...

Observation and operator log

Record task, user action, operator action, timing, help and outcome separately.

...

Debriefing and learning decision

What was disclosed, how did the person respond and what limits remain?

...

Answer the prompts with your own information. Explicitly mark unknowns and assumptions.

NN/g: The Wizard of Oz Method in UX

Simulation: Simulation

Expected artifact: Simulation contract

Is it clear which assumption the test can answer?

Entry:

...

Is it clear which assumption the test can answer?

Protocol: Protocol

Expected artifact: Execution protocol

Are data access, withdrawal and debriefing clear?

Entry:

...

Are data access, withdrawal and debriefing clear?

Rehearsal: Rehearsal

Expected artifact: Rehearsed setup

Is operation reproducible rather than improvised?

Entry:

...

Is operation reproducible rather than improvised?

Observation: Observation

Expected artifact: Behaviour log

Is assistance visible as an intervention in the log?

Entry:

...

Is assistance visible as an intervention in the log?

Debriefing: Debriefing

Expected artifact: Test findings

Have the simulation and its limits been disclosed?

Entry:

...

Have the simulation and its limits been disclosed?

Open questions and next steps

...

Method guide: https://methodatlas.meierhoff-systems.de/en/methods/wizard-of-oz-test/run-sheet

Example output

Concrete filled scenario, fictional example

Fictional example for orientation. Counts and timings describe this case and are not universal requirements.

wizard-of-oz-test-beispiel.md

Concrete filled scenario, fictional example

Fictional example

Hypothesis and facade: Fictional: a course assistant suggests classes matching three stated time windows. Hypothesis: people can then choose a suitable option independently. A concealed operator selects from a fixed course list; no real AI exists.

Operator rules and limits: Lea returns at most three matching offers within 20–40 seconds. No match triggers a request for another time window. Unplanned responses are logged as deviations.

Participation and debriefing: Voluntary participation in a research prototype; only fictional scheduling data, human access to inputs disclosed. The exact operator workflow is explained immediately afterwards; withdrawal and agreed deletion are available.

Observation and operator log: P02 enters three times; Lea responds after 32 seconds. P02 overlooks location and chooses a time-compatible class at the wrong venue. The moderator explains only after task completion.

Debriefing and learning decision: P02 learns that Lea selected the options. The team adds location comparison and retests. Automated quality, scaling and operating costs are not established by this manual trial.