From the Curtain to Kansas: Conducting Wizard-of-Oz Studies in the Wild

Conversational agents have been touted for their potential to support individuals over time as health coaches or personal assistants, but have yet to live up to this potential. Wizard-of-oz (WOz) methods enable researchers to test early prototypes of conversational applications before they are fully implemented, with a human "wizard" filling in the gaps in functionality. Current WOz methods, however, are more commonly used for studies in a lab setting, rather than deployment studies, which more accurately capture users' interactions in-the-wild. We argue for the need for WOz methods for deployment studies that address key challenges, namely the need for easy-to-prototype technology that works reliably in the wild, as well as the user's expectations for 24/7 availability. We describe an initial approach that begins to address these challenges, as well as the insights gleaned from a two-week WOz study of t2.coach, a conversational agent health coach for diabetes self-management. We argue that the findings from our WOz study could not have been identified from a lab usability study with the same prototype, and the need for the research community to further develop methods for WOz deployment studies.