Improving the reliability and validity of wizard-of-oz methods

Wizard-of-Oz (WoZ) is a flexible, efficient and cost-economic method to the design and evaluation of interaction systems, particularly such of natural dialogue and smart systems. However, the literature review in the beginning of this research indicated that researchers struggled to implement WoZ and be able to gain reliable and valid experimental results in terms of system facilitation consistency; and WoZ has been criticised for a lack of systematic assessment of influence variables, especially when it was used to study new emerging information and communication technologies. Hence, this research aimed to understand and improve the reliability and validity of WoZ.
The research consisted of a series of empirical studies to incrementally deepen the understanding of influence variables. The main body of research comprised studies investigating (1) the impact of schema as WoZ study guidelines, (2) the impact of control panel in system facilitation, (3) the variables affecting evaluator’s interpretation of schema, control panel and subject activity, and (4) the differences in multiple evaluators’ system facilitation.
The results indicated that neither rigorous nor general schemas supported highly reliable system facilitation; rather, schemas should be accordingly proposed on the base of predictable or unpredictable user interactions. Also the results revealed the hidden relationships between control panel and system facilitation through identifying the control panel influence factors such like layouts and functions and their connections with system facilitation. Additionally, despite the difficulty of synchronising evaluators’ individual expertise and experiences was admitted, the research findings suggested practical measurements to address the individual differences at acceptable levels through applying additional assistance and constraints to evaluator’s system facilitation judgement and execution. And the results also provided secondary understanding towards smart system design for domestic communication and the development of WoZ system.