Facial Emotion Recognition Using Transfer Learning in the Deep CNN

被引:98
作者
Akhand, M. A. H. [1 ]
Roy, Shuvendu [1 ]
Siddique, Nazmul [2 ]
Kamal, Md Abdus Samad [3 ]
Shimamura, Tetsuya [4 ]
机构
[1] Khulna Univ Engn & Technol, Dept Comp Sci & Engn, Khulna 9203, Bangladesh
[2] Ulster Univ, Sch Comp Engn & Intelligent Syst, Coleraine BT48 7JL, Londonderry, North Ireland
[3] Gunma Univ, Grad Sch Sci & Technol, Kiryu, Gumma 3768515, Japan
[4] Saitama Univ, Grad Sch Sci & Engn, Saitama 3388570, Japan
关键词
convolutional neural network (CNN); deep CNN; emotion recognition; transfer learning; CONVOLUTIONAL NEURAL-NETWORKS; EXPRESSION RECOGNITION; ARCHITECTURES; FACE;
D O I
10.3390/electronics10091036
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Human facial emotion recognition (FER) has attracted the attention of the research community for its promising applications. Mapping different facial expressions to the respective emotional states are the main task in FER. The classical FER consists of two major steps: feature extraction and emotion recognition. Currently, the Deep Neural Networks, especially the Convolutional Neural Network (CNN), is widely used in FER by virtue of its inherent feature extraction mechanism from images. Several works have been reported on CNN with only a few layers to resolve FER problems. However, standard shallow CNNs with straightforward learning schemes have limited feature extraction capability to capture emotion information from high-resolution images. A notable drawback of the most existing methods is that they consider only the frontal images (i.e., ignore profile views for convenience), although the profile views taken from different angles are important for a practical FER system. For developing a highly accurate FER system, this study proposes a very Deep CNN (DCNN) modeling through Transfer Learning (TL) technique where a pre-trained DCNN model is adopted by replacing its dense upper layer(s) compatible with FER, and the model is fine-tuned with facial emotion data. A novel pipeline strategy is introduced, where the training of the dense layer(s) is followed by tuning each of the pre-trained DCNN blocks successively that has led to gradual improvement of the accuracy of FER to a higher level. The proposed FER system is verified on eight different pre-trained DCNN models (VGG-16, VGG-19, ResNet-18, ResNet-34, ResNet-50, ResNet-152, Inception-v3 and DenseNet-161) and well-known KDEF and JAFFE facial image datasets. FER is very challenging even for frontal views alone. FER on the KDEF dataset poses further challenges due to the diversity of images with different profile views together with frontal views. The proposed method achieved remarkable accuracy on both datasets with pre-trained models. On a 10-fold cross-validation way, the best achieved FER accuracies with DenseNet-161 on test sets of KDEF and JAFFE are 96.51% and 99.52%, respectively. The evaluation results reveal the superiority of the proposed FER system over the existing ones regarding emotion detection accuracy. Moreover, the achieved performance on the KDEF dataset with profile views is promising as it clearly demonstrates the required proficiency for real-life applications.
引用
收藏
页数:19
相关论文
共 50 条
  • [21] Facial emotion recognition using geometrical features based deep learning techniques
    Iqbal, J. L. Mazher
    Kumar, M. Senthil
    Mishra, Geetishree
    Asha, G. R.
    Saritha, A. N.
    Karthik, A.
    BonthuKotaiah, N.
    INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL, 2023, 18 (04)
  • [22] Facial Expression Recognition using Transfer Learning
    Ramalingam, Soodamani
    Garzia, Fabio
    2018 52ND ANNUAL IEEE INTERNATIONAL CARNAHAN CONFERENCE ON SECURITY TECHNOLOGY (ICCST), 2018, : 152 - 156
  • [23] Learning deep features to recognise speech emotion using merged deep CNN
    Zhao, Jianfeng
    Mao, Xia
    Chen, Lijiang
    IET SIGNAL PROCESSING, 2018, 12 (06) : 713 - 721
  • [24] Time efficient real time facial expression recognition with CNN and transfer learning
    Podder, Tanusree
    Bhattacharya, Diptendu
    Majumdar, Abhishek
    SADHANA-ACADEMY PROCEEDINGS IN ENGINEERING SCIENCES, 2022, 47 (03):
  • [25] Facial Expression Recognition with Identity and Emotion Joint Learning
    Li, Ming
    Xu, Hao
    Huang, Xingchang
    Song, Zhanmei
    Liu, Xiaolin
    Li, Xin
    IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2021, 12 (02) : 544 - 550
  • [26] EMOTION RECOGNITION USING DEEP LEARNING
    Priya, R. N. Beena
    Hanmandlu, M.
    Vasikarla, Shantaram
    2021 IEEE APPLIED IMAGERY PATTERN RECOGNITION WORKSHOP (AIPR), 2021,
  • [27] Facial Expression Recognition Using Transfer Learning on Deep Convolutional Network
    Hablani, Ramchand
    BIOSCIENCE BIOTECHNOLOGY RESEARCH COMMUNICATIONS, 2020, 13 (14): : 185 - 188
  • [28] Enhanced Deep Learning Hybrid Model of CNN Based on Spatial Transformer Network for Facial Expression Recognition
    Khan, Nizamuddin
    Singh, Ajay Vikram
    Agrawal, Rajeev
    INTERNATIONAL JOURNAL OF PATTERN RECOGNITION AND ARTIFICIAL INTELLIGENCE, 2022, 36 (14)
  • [29] A Deep CNN Approach with Transfer Learning for Image Recognition
    Iorga, Cristian
    Neagoe, Victor-Emil
    PROCEEDINGS OF THE 11TH INTERNATIONAL CONFERENCE ON ELECTRONICS, COMPUTERS AND ARTIFICIAL INTELLIGENCE (ECAI-2019), 2019,
  • [30] Improvement of emotion recognition from facial images using deep learning and early stopping cross validation
    Bentoumi, Mohamed
    Daoud, Mohamed
    Benaouali, Mohamed
    Ahmed, Abdelmalik Taleb
    MULTIMEDIA TOOLS AND APPLICATIONS, 2022, 81 (21) : 29887 - 29917