Two-Step Classification Method for Sadness and Fear Facial Expression Classification Using Facial Feature Points and FACS

Although systems have been developed to automatically estimate the evaluation of a work (e.g., comics, movie, etc.) based on the facial expressions of people viewing the work, they are insufficient for estimating the evaluation of moving or horror works, for which sadness and fear are less likely to be expressed in facial expressions and these expressions are thought to lead to evaluation. In this paper, we propose a new method to improve the accuracy of classification of facial expressions of sadness and fear. Facial feature points and ActionUnits (AUs) are extracted from facial images to set facial features and classify facial images into seven facial expressions: six basic facial expressions (anger, disgust, fear, happiness, sadness, and surprise) and neutral. First, the expressions of anger and sadness, which are easily confused, are merged into a single category, and similarly fear and surprise are merged to convert seven categories into five. Then, two-step classificationwas performed by reclassifying each of themerged facial expressions into two categories. Furthermore, the importance of each feature was compared, and the feature most suitable for each classification step was used to improve the classification accuracy. The results of a random forest model classification experiment using 490 face images from the Karolinska Directed Emotional Faces (KDEF) dataset showed that two-step classification performed better than one-step classification in the classification of sadness and fear. The accuracy was further improved by carefully selecting features.