A shared realities workflow for interactive design using virtual reality and three-dimensional depth sensing

This article presents the ongoing development and testing of a "shared realities" computational workflow to support iterative user-centered design with an interactive system. The broader aim is to address the challenges associated with observing and recording user interactions within the context of use for improving the performance of an interactive system. A museum installation is used as an initial test bed to validate the following hypothesis: by integrating three-dimensional depth sensing and virtual reality for interaction design and user behavior observations, the shared realities workflow provides an iterative feedback loop that allows for remote observations and recordings for faster and effective decision-making. The methods presented focus on the software development for gestural interaction and user point cloud observations, as well as the integration of virtual reality tools for iterative design of the interface and system performance assessment. Experimental testing demonstrates viability of the shared realities workflow for observing and recording user interaction behaviors and evaluating system performance. Contributions to computational design, technical challenges, and ethical considerations are discussed, as well as directions for future work.