A hybrid deep learning approach for online learners' recognition using screen name, and learners' emotion detection

Online learning environments (OLEs) have enabled global access to education during and after the COVID-19 pandemic. While widely accepted, OLEs pose challenges for teachers in recognising students and understanding their emotions, unlike in traditional classrooms. To address this, a hybrid deep learning architecture is proposed that uses screen names and facial expressions to identify students and detect their emotional states. The MTCNN model is used to segment the frames of an online class video where each segmented part contains a student's face and screen name. The EAST and Py-Tesseract and face emotion recognition models are then applied on these segmented parts to extract screen names and detect the emotional states of the students. This helps in detecting individual student's emotional feedback and overall class feedback. Finally, the students are classified as engaged or not engaged. The proposed architecture will make OLEs as efficient as traditional classrooms.