Incorporating real-world object into virtual reality: using mobile device input with augmented virtuality

Interactive virtual reality (VR) experience has become more common and widespread these days. The interaction is enabled by capturing inputs from users usually by mechanical hand-held controllers. Mainstream head-mounted display (HMD) based VR platforms come with these trackable multi-DOF controllers with a high level of versatility. This enables a tremendous diversity in input types and forms to provide more interactive and immersive VR experience. However, these devices lack in cross-platform compatibility as the protocols are non-standardized. Thus, it leads to a steep learning curve. The trouble in learning these devices results in non-intuitiveness and confusion to users. Also, the VR isolation may cause problems since the platforms are not designed to reflect any real environment around the user. In this paper, we investigate an alternate input method for virtual reality by incorporating a real-world object with augmented virtuality. We set a mobile phone as the real-world object for this phase of the study. We import a mobile phone from the real world and evaluate the possibility of utilizing it as an input device. The objective of this method is to give VR users more flexibility in input methods for VR and to reduce the physical barrier between the reality and virtual environment by using augmented virtuality. We propose a method for VR that overlays both real texture of a mobile phone and user's hand onto a virtual scene in real-time by utilizing external cameras attached on the HMD. While being immersed in virtual environment with the HMD, users can manipulate the device naturally for entering inputs to alter the virtual environment without removing the headset. We implement our prototype in two phases to explore its possibilities as a new input technique for VR. The evaluation was conducted by setting up different kinds of conditions and tasks in virtual environment. The effectiveness of the proposed method is verified experimentally in specific given tasks in comparison with conventional controllers. Furthermore, we address and improve important points for accurate real image display in the virtual environment which we have found in the process of the implementation and the experiment. The outcome of the study indicates that our approach based on augmented virtuality, which is a relatively new approach for mixed reality, could lead to a path for an effective and natural input method for VR.