Classification of motor errors to provide real-time feedback for sports coaching in virtual reality - A case study in squats and Tai Chi pushes

For successful fitness coaching in virtual reality, movements of a trainee must be analyzed in order to provide feedback. To date, most coaching systems only provide coarse information on movement quality. We propose a novel pipeline to detect a trainee's errors during exercise that is designed to automatically generate feedback for the trainee. Our pipeline consists of an online temporal warp of a trainee's motion, followed by Random-Forest-based feature selection. The selected features are used for the classification performed by Support Vector Machines. Our feedback to the trainee can consist of predefined verbal information as well as automatically generated visual augmentations. For the latter, we exploit information on feature importance to generate real-time feedback in terms of augmented color highlights on the trainee's avatar. We show our pipeline's superiority over two popular approaches from human activity recognition applied to our problem, k-Nearest Neighbor, combined with Dynamic Time Warping (KNN-DTW), as well as a recent combination of Convolutional Neural Networks with a Long Short-term Memory Network. We compare classification quality, time needed for classification, as well as the classifiers' ability to automatically generate augmented feedback. In an exemplary application, we demonstrate that our pipeline is suitable to deliver verbal as well as automatically generated augmented feedback inside a CAVE-based sports training environment in virtual reality. (C) 2018 Elsevier Ltd. All rights reserved.