FERMOUTH: Facial Emotion Recognition from the MOUTH Region

People use various nonverbal communicative channels to convey emotions, among which facial expressions are considered the most important ones. Consequently, automatic Facial Expression Recognition (FER) is a crucial task for enhancing computers' perceptive abilities, particularly in human-computer interaction. Although state-of-the-art FER systems can identify emotions from the entire face, situations may arise where occlusions prevent the entire face from being visible. During the COVID-19 pandemic, many FER systems have been developed for recognizing emotions from the eye region due to the obligation to wear a mask. However, in many situations, the eyes may be covered, for instance, by sunglasses or virtual reality devices. In this paper, we faced the problem of developing a FER system that solely considers the mouth region and classifies emotions using only the lower part of the face. We tested the effectiveness of this FER system in recognizing emotions from the lower part of the face and compared the results to a FER system trained on the same datasets using the same approach on the entire face. As expected, emotions primarily associated with the mouth region (e.g., happiness, surprise) were recognized with minimal loss compared to the entire face. Nevertheless, even though most negative emotions were not accurately detected using only the mouth region, in cases where the face is partially covered, this area may still provide some information about the displayed emotion.