Remote photoplethysmography (rPPG) based learning fatigue detection

Remote photoplethysmography (rPPG), which uses a facial video to measure skin reflection variations, is a contactless method formonitoring human cardiovascular activity. Due to its simplicity, convenience and potential in large-scale application, rPPG has gained more attention over the decade. However, the accuracy, reliability, and computational complexity have not reached the expected standards, thus rPPG has a very limited application in the educational field. In order to alleviate this issue, this study proposes an rPPG-based learning fatigue detection system, which consists of the following three modules. First, we propose an rPPG extraction module, which realizes real-time pervasive biomedical signal monitoring. Second, we propose an rPPG reconstruction module, which evaluates heart rate using a hybrid of 1D and 2D deep convolutional neural network approach. Third, we propose a learning fatigue classification module based on multi-source feature fusion, which classifies a learner's state into non-fatigue and fatigue. In order to verify the performance, the proposed system is tested on a self-collected dataset. Experimental results demonstrate that (i) the accuracy of heart rate evaluation is better than the cutting-edge methods; and (ii) based on both the subject-dependent and independent cross validations, the proposed system succeeded in not only learning person-independent features for fatigue detection but also detecting early fatigue with very high accuracy.