A two-stage multimodal emotion analysis using body actions and facial features

This study uses a combination of body action detection and facial expression identification to analyze emotions across various positive and negative scales. The body action dataset includes annotated actions from some well-known databases and a self-created series of coordinate movements. At the first stage, neural network models such as CNN + LSTM, LSTM, and Action Transformer are used to train on these datasets, along with various attention mechanisms to confirm their efficacy. Facial expression identification is enhanced using open-source datasets and the DeepFace toolkit developed by the Meta AI Research Group. At the second stage, the emotional analysis is conducted by combining the outcomes of body action recognition and facial expression identification and then fed into kNN, SVM, or decision tree for positive and negative analysis. The experimental outcomes highlight several crucial points. The accuracy rates for action recognition using RNN + LSTM and Action Transformer are 81.1% and 88.4%, respectively. Moreover, DeepFace attains a facial expression identification accuracy of 97%. When consolidated for emotional analysis using kNN, the test accuracy improves significantly, showcasing its strong performance in various scenarios.