Complete firstSolution Interview
The solution concept has been discussed with target users so the experience to test is roughly validated.
Without: Without prior validation, we simulate an experience nobody needs and manual work is wasted.
RunsheetProduct Discovery
Wizard of Oz Test
ComplexityHigh
Time2-10 Tage
Participants5-20 Nutzer
FormatWorkshop + async
MaturityEstablished
01
Prerequisite
What needs to be finished first
02
Preparation
What needs to be ready before start
Materials
Working frontend prototype (click dummy or simple web frontend); backend scripts or manual operators (wizards); logging setup; SLA plan for response times; ethical briefing for test participants.
People / roles
One test owner defining experience and coordinating wizards; two to five wizards manually responding; UX researcher for observation; test users.
Pre-read
Experience to test in one sentence; SLA expectation; script for typical requests; escalation rule for unclear cases; disclosure about manual involvement after the test.
Time needed
2-10 days
Setup
Build frontend so requests go to a wizard channel. Wizards work in shifts with clear response rules. Log all requests and responses. Inform users after the test about manual involvement.
03
Core question
The one question this method answers
Does the simulated automatic experience deliver recognizable value to users, and which tasks are realistically automatable?
04
Flow
Marker: Phase
| Step | Duration | Action | Hint |
|---|---|---|---|
1Phase 1: Define experience and script | Setup, 1-2 days | Describe the experience as end-to-end flow. Define typical requests and answers per step. Set SLA, tone, and escalation path. | If the script does not cover 70% of requests on day one, the experience is too broad. Narrow scope. |
2Phase 2: Frontend and wizard setup | Setup, 1-3 days | Build frontend (chat, form, app stub). Wizard dashboard with incoming request queue. Shift plan for wizards. Test with 2-3 internal participants. | Wizards need script training. Live improvisation creates inconsistent experience. |
3Phase 3: Live with real test users | Runtime, 5-7 days | Invite test users. Wizards work shifts and keep SLA. UX researcher observes patterns. Daily wizard sync for consistency. | If SLA breaks, learning is distorted. Fewer users with stable experience beat many users with chaos. |
4Phase 4: Disclosure and analysis | Synthesis, 1-2 days | After test end, disclose manual involvement. Interview or survey users. Analyze logs: frequent requests, wizard effort per request, escalated cases. | Disclosure is an ethical duty, not optional. |
05
Artifact
What comes out at the end
Form
Experiment report with experience description, logs of requests and answers, wizard effort per request, usage patterns, user feedback after disclosure, scaling risks, and recommendation for Build, Iterate, or Kill.
Versioning / ownership
Own report per iteration with date, scope, and wizard setup. Document script updates as delta.
Tool alternatives
- Intercom or Crisp as chat frontend
- Slack as wizard dashboard with bot binding
- Airtable for request queue
- Custom webhook plus Notion for logs
wizard-of-oz-test-working-template.md
Compact working template for Wizard of Oz Test with context, input, output artifacts, and next step.
Wizard of Oz Test Working Template
Goal
Manually simulates automated product behavior behind the scenes.
Context
When and for what do we use this method?
Input
Which data, observations, decisions, or materials are available?
Execution
Short notes along the runsheet.
Output artifacts
- Prototype Learnings:
- Manual Effort Notes:
- Risk List:
Assumptions and open questions
- ...
Decision / Next step
Owner, date, and success signal.
06
Example output
Concrete filled scenario, fictional example
wizard-of-oz-test-beispiel.md
Concrete filled scenario, fictional example
Wizard of Oz - AI booking assistant (2026-05-12 to 2026-05-22)
Experience: User submits receipts as photo. Assistant categorizes, suggests account, and flags anomalies. Response under 2 min.
Setup: Mobile web frontend with upload. Requests land in Slack channel #wizard-queue. Three wizards in shifts Mon-Fri 9-18. Script with 80 typical booking cases.
Results (5 users, 7 days, 247 receipts):
- Wizard effort: average 1 min 50 s per receipt, median 80 s.
- Escalation rate: 14%.
- User feedback: 4 of 5 rate experience helpful.
Recommendation: Build. Script covers 86% of cases, automation with regex and ML realistic.
07
Pitfalls
Recognize symptoms and steer against them
Trap
Wizards improvise
Symptom
Answers vary heavily, user experience inconsistent.
What to do
Script training before live phase. Daily sync. Extend script for frequent escalations.
Trap
SLA breaks
Symptom
Response times vary between 30 s and 30 min.
What to do
Check wizard capacity against request load. Reduce user count if overloaded.
Trap
No disclosure
Symptom
Users believe the solution is already real and AI-driven.
What to do
Disclosure after test end is mandatory.
Trap
Scope too broad
Symptom
Script tries to cover all use cases; wizards cannot learn it.
What to do
Limit to one end-to-end flow.
Trap
Logs missing or unstructured
Symptom
Analysis barely possible, wizard effort guessed.
What to do
Structured logs per request: timestamp, type, answer, escalation, effort.
08
Stop criteria
Done signals checkable in under a minute
Solution hypothesis not validated upfront through interviews.
Wizard capacity insufficient for expected load, SLA not maintainable.
No disclosure planned, ethical duty violated.
Test would start without script, consistency missing.
Test setup under 3 days runtime, patterns not visible.
Scaling would be regulatorily problematic, test misleading.
Finished the runsheet?
Go to the profile for purpose, similar methods, and sources or continue to the next method in the catalog.