Automatic Detection of Irony Based on Acoustic Features and Facial Expressions

被引:0
作者
Kochetkova, Uliana [1 ]
Slcreline, Pavel [1 ]
Evdokimova, Vera [1 ]
Borisov, Nikolai [1 ]
Scherbakov, Pavel [1 ]
Fedkin, Petr [1 ]
German, Rada [1 ]
机构
[1] St Petersburg State Univ, 7-9 Univ Skaya Embankment, St Petersburg, Russia
来源
SPEECH AND COMPUTER, SPECOM 2024, PT II | 2025年 / 15300卷
关键词
Irony; Multimedia Speech Corpus; Artificial Neural Networks; Acoustic Feature Extraction; Facial Expression Analysis; FACE;
D O I
10.1007/978-3-031-78014-1_6
中图分类号
O42 [声学];
学科分类号
070206 ; 082403 ;
摘要
The current study deals with the automatic analysis of verbal irony using artificial neural networks. Detection of verbal irony is an important task nowadays, because the effectiveness of the communication depends on the correct interpretation of sentences with an ambiguous meaning. In the case, when the context is lacking, the correct sense can be understood not from the lexical content, but through phonetic features, as well as through co-speech mimics and gestures. Thus we accomplished a new research on the material of the multimedia corpus of Russian ironic speech, which contains the detailed phonetic annotation and irony evaluation by native listeners in perceptual auditory experiments. Two types of automated analysis were accomplished: based on acoustic feature and facial expression extraction. The use of the fully connected neural network and of the Wav2Vec 2.0 model for the automatic irony detection in audio signal demonstrated high level of irony recognition. We also tested on a part of the corpus the recognition of ironic facial expressions in video signal using convolutional neural network and the PyFeat library, which allowed us to conclude that this model can give good results when we increase the amount of the material.
引用
收藏
页码:70 / 82
页数:13
相关论文
共 50 条
[21]   Recognition of emotions from video using acoustic and facial features [J].
Rao, K. Sreenivasa ;
Koolagudi, Shashidhar G. .
SIGNAL IMAGE AND VIDEO PROCESSING, 2015, 9 (05) :1029-1045
[22]   An observation based muscle model for simulation of facial expressions [J].
Erkoc, Tugba ;
Agdogan, Didem ;
Eskil, M. Taner .
SIGNAL PROCESSING-IMAGE COMMUNICATION, 2018, 64 :11-20
[23]   DEEP BINARY REPRESENTATION OF FACIAL EXPRESSIONS: A NOVEL FRAMEWORK FOR AUTOMATIC PAIN INTENSITY RECOGNITION [J].
Tavakolian, Mohammad ;
Hadid, Abdenour .
2018 25TH IEEE INTERNATIONAL CONFERENCE ON IMAGE PROCESSING (ICIP), 2018, :1952-1956
[24]   Pain detection from facial expressions using domain adaptation technique [J].
Rathee, Neeru ;
Pahal, Sudesh ;
Sheoran, Poonam .
PATTERN ANALYSIS AND APPLICATIONS, 2022, 25 (03) :567-574
[25]   Eye Landmarks Detection Technology for Facial Micro-Expressions Analysis [J].
Sergeeva, Anna D. ;
Sablina, Victoria A. .
2020 9TH MEDITERRANEAN CONFERENCE ON EMBEDDED COMPUTING (MECO), 2020, :443-446
[26]   Pain detection from facial expressions using domain adaptation technique [J].
Neeru Rathee ;
Sudesh Pahal ;
Poonam Sheoran .
Pattern Analysis and Applications, 2022, 25 :567-574
[27]   THE EXTRACTION OF ACOUSTIC FEATURES OF INFANT CRY FOR EMOTION DETECTION BASED ON PITCH AND FORMANTS [J].
Hidayati, Rahmat ;
Purnama, I. Ketut Eddy ;
Purnomo, Mauridhi Hery .
ICICI-BME: 2009 INTERNATIONAL CONFERENCE ON INSTRUMENTATION, COMMUNICATION, INFORMATION TECHNOLOGY, AND BIOMEDICAL ENGINEERING, 2009, :328-332
[28]   A review of image-based automatic facial landmark identification techniques [J].
Benjamin Johnston ;
Philip de Chazal .
EURASIP Journal on Image and Video Processing, 2018
[29]   A review of image-based automatic facial landmark identification techniques [J].
Johnston, Benjamin ;
de Chazal, Philip .
EURASIP JOURNAL ON IMAGE AND VIDEO PROCESSING, 2018,
[30]   A transformer-based approach to irony and sarcasm detection [J].
Rolandos Alexandros Potamias ;
Georgios Siolas ;
Andreas - Georgios Stafylopatis .
Neural Computing and Applications, 2020, 32 :17309-17320