Object Detection with Deep Learning for a Virtual Reality Based Training Simulator

Virtual Reality (VR) provides immersive user experience which makes them a cost effective solution to employ for various training purposes. However, a major shortcoming of VR systems is their limitation when it comes to interacting with the environment. Typically, when users wear a head mounted display their vision will be limited to virtual world and their external vision will be blocked. They will not be able to see useful objects in their environment such as controllers, buttons or even their hands. In this paper, we describe design of a training system for aerospace industry where real and virtual images blended, creating an augmented virtuality. The real world images are obtained from a camera mounted on the head-mounted-display. Some of the predefined objects, such as game controllers and user's hands, are detected via deep learning algorithms and blended into the virtual reality images providing a more comfortable and immersive user experience. Furthermore, camera and object detection algorithms are employed to interact with VR headset making it more convenient tool for training simulators.