The Acoustically Emotion-Aware Conversational Agent With Speech Emotion Recognition and Empathetic Responses

被引：7
作者：
Hu, Jiaxiong [1 ]
Huang, Yun [2 ]
Hu, Xiaozhu [3 ]
Xu, Yingqing [1 ]
机构：
[1] Tsinghua Univ, Acad Arts & Design, Beijing 100084, Peoples R China
[2] Univ Illinois, Sch Informat Sci, Champaign, IL 61820 USA
[3] Tsinghua Univ, Dept Comp Sci & Technol, Beijing 100084, Peoples R China
来源：
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING | 2023年 / 14卷 / 01期
基金：
中国国家自然科学基金;
关键词：
Emotion recognition; Speech recognition; Databases; Sentiment analysis; Games; Electronic mail; Convolutional neural networks; Human-centered computing; emotion in human-computer interaction; influencing human emotional states; intelligent agents;
D O I：
10.1109/TAFFC.2022.3205919
中图分类号：
TP18 [人工智能理论];
学科分类号：
081104 ; 0812 ; 0835 ; 1405 ;
摘要：
Emotion is important for the conversational user interface. In prior research, conversational agents (CAs) employ natural language process techniques to create affective interaction based on text. However, the use of acoustic features of speech for voice-based CAs is under exploration. This work presents an acoustically emotion-aware CA that enables speech emotion recognition and stylizes responses with empathetic feedback and interjections. We conducted an experiment with 75 participants to evaluate their perceived emotional intelligence (PEI) after interacting with the CA. Our results show that the acoustical emotion-awareness increased the participants' PEI of the CA, and the empathetic responses from the CA helped alleviate some participants' negative emotions. Our work provides implications for designing future CAs with better PEI.
引用
收藏
页码：17 / 30
页数：14
相关论文
共 50 条
[21]   English speech emotion recognition method based on speech recognition [J].
Man Liu .
International Journal of Speech Technology, 2022, 25 :391-398
[22]   Multi-Domain Emotion Recognition Enhancement: A Novel Domain Adaptation Technique for Speech-Emotion Recognition [J].
Amjad, Ammar ;
Khuntia, Sucharita ;
Chang, Hsien-Tsung ;
Tai, Li-Chia .
IEEE TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2025, 33 :528-541
[23]   Differential Impacts of Monologue and Conversation on Speech Emotion Recognition [J].
Chien, Woan-Shiuan ;
Upadhyay, Shreya G. ;
Lin, Wei-Cheng ;
Busso, Carlos ;
Lee, Chi-Chun .
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2025, 16 (02) :485-498
[24]   Emotion-Aware Control Framework for Human-Robot Collaboration [J].
Pupa, Andrea ;
Varrecchia, Raffaele ;
Secchi, Cristian .
HUMAN-FRIENDLY ROBOTICS 2023, HFR 2023, 2024, 29 :66-78
[25]   Enhancing Speech Emotion Recognition With Conditional Emotion Feature Diffusion and Progressive Interleaved Learning Strategy [J].
Liu, Yang ;
Chen, Xin ;
Peng, Zhichao ;
Li, Yongwei ;
Li, Xingfeng ;
Song, Peng ;
Unoki, Masashi ;
Zhao, Zhen .
IEEE TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2025, 33 :1787-1800
[26]   Multiroom Speech Emotion Recognition [J].
Shalev, Erez ;
Cohen, Israel .
2022 30TH EUROPEAN SIGNAL PROCESSING CONFERENCE (EUSIPCO 2022), 2022, :135-139
[27]   Persian Speech Emotion Recognition [J].
Savargiv, Mohammad ;
Bastanfard, Azam .
2015 7TH CONFERENCE ON INFORMATION AND KNOWLEDGE TECHNOLOGY (IKT), 2015,
[28]   MahaEmoSen: Towards Emotion-aware Multimodal Marathi Sentiment Analysis [J].
Chaudhari, Prasad ;
Nandeshwar, Pankaj ;
Bansal, Shubhi ;
Kumar, Nagendra .
ACM TRANSACTIONS ON ASIAN AND LOW-RESOURCE LANGUAGE INFORMATION PROCESSING, 2023, 22 (09)
[29]   An emotion-aware search engine for multimedia content based on deep learning algorithms [J].
Chiorrini, Andrea ;
Diamantini, Claudia ;
Mircoli, Alex ;
Potena, Domenico ;
Storti, Emanuele .
INTERNATIONAL JOURNAL OF COMPUTER APPLICATIONS IN TECHNOLOGY, 2023, 73 (02) :130-139
[30]   SENTIMENT-AWARE AUTOMATIC SPEECH RECOGNITION PRE-TRAINING FOR ENHANCED SPEECH EMOTION RECOGNITION [J].
Ghriss, Ayoub ;
Yang, Bo ;
Rozgic, Viktor ;
Shriberg, Elizabeth ;
Wang, Chao .
2022 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2022, :7347-7351
12345