Automated Usability Evaluation of Virtual Reality Applications

Virtual reality (VR) and VR applications have reached the end-user and, hence, the demands on usability, also for novel applications, have increased. This situation requires VR usability evaluation methods that can be applied quickly, even after a first release of an application. In this article, we describe such an approach, which is fully automated and does not ask users to perform predefined tasks in a fixed test setting. Instead, it works on recordings of the actual usage of a VR application from which it generates task trees. Afterwards, it analyzes these task trees to search for usability smells, i.e., user behavior indicating usability issues. Our approach provides detailed descriptions of the usability issues that have been found and how they can be solved. We performed a large case study to evaluate our approach and show that it is capable of correctly identifying usability issues. Although our approach is applicable for different VR interaction modalities, such as gaze, controller, or hand interaction, it also has limitations. For example, it can detect diverse issues related to user efficiency, but specific misunderstandings of users cannot be uncovered.