EXController: Enhancing Interaction Capability for VR Handheld Controllers using Real-Time Vision Sensing

This paper presents EXController, a new controller-mounted finger posture recognition device specially designed for VR handheld controllers. We seek to provide additional input through real-time visions sensing by attaching a near infrared (NIR) camera onto the controller. We designed and implemented an exploratory prototype with a HTC Vive controller. The NIR camera is modified from a traditional webcam and applied with a data-driven Convolutional Neural Network (CNN) classifier. We designed 12 different finger gestures and trained the CNN classifier with a dataset from 20 subjects, achieving an average accuracy of 86.17% across subjects, and, approximately snore than 92% on three oldie linger postures, and more than 89%. on the top-4 accuracy postures. We also developed a Unity demo that shows matched finger animations. miming at approximately 27 fps in real-time.