A Real-time Multimodal Intelligent Tutoring Emotion Recognition System (MITERS)

Emotion recognition can be used in a wide range of applications. We are interested in the E-learning system because of the several benefits of learning anywhere and anytime. Despite important advances, students' emotions can influence the learning process, and this is the case with both traditional learning as well as E-learning. Emotion can limit and blocks our ability to learn, think, and solve problems. On the contrary, it can drive us to success by boosting our innate mental ability when we love what we do and when we are affected by happiness and excitement. In recent years, a large number of studies have addressed the problem of emotion recognition based on different modalities. But the information provided by student emotion recognition based on single-modal data like face is insufficient. Additionally, selecting one affective state over another might be challenging at times. To remove these ambiguities, we propose an Intelligent Affective Tutoring System named "Multimodal Intelligent Tutoring Emotion Recognition System" MITERS that merges three modalities such as face, text, and speech simultaneously. Our System is a real-time-based system that detects the emotion of students and gives adequate feedback. For this purpose, we use deep learning techniques. Among these are (a) Deep Convolution Neural Network (DCNN) used to detect emotion from face modality, (b) Bidirectional Long Short Memory (BiLSTM) for predicting emotions from text information, and (c) Convolutional Neural Network (CNN) which is used to detect emotions from speech modality. The experimental results are compared with some of the well-known approaches and the proposed MITERS has performed well with a classification accuracy of 97% in MELD which is a multimodal database.