Speech Emotion Recognition among Couples using the Peak-End Rule and Transfer Learning

被引:5
作者
Boateng, George [1 ]
Sels, Laura [2 ]
Kuppens, Peter [3 ]
Hilpert, Peter [4 ]
Kowatsch, Tobias [1 ]
机构
[1] Swiss Fed Inst Technol, Zurich, Switzerland
[2] Univ Ghent, Ghent, Belgium
[3] Katholieke Univ Leuven, Leuven, Belgium
[4] Univ Surrey, Surrey, England
来源
COMPANION PUBLICATON OF THE 2020 INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION (ICMI '20 COMPANION) | 2020年
关键词
Speech emotion recognition; Speech processing; Affective computing; Couples; Transfer Learning; Peak-end rule; Convolutional neural network; Support vector machine; MODEL;
D O I
10.1145/3395035.3425253
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
Extensive couples' literature shows that how couples feel after a conflict is predicted by certain emotional aspects of that conversation. Understanding the emotions of couples leads to a better understanding of partners' mental well-being and consequently their relationships. Hence, automatic emotion recognition among couples could potentially guide interventions to help couples improve their emotional well-being and their relationships. It has been shown that people's global emotional judgment after an experience is strongly influenced by the emotional extremes and ending of that experience, known as the peak-end rule. In this work, we leveraged this theory and used machine learning to investigate, which audio segments can be used to best predict the end-of-conversation emotions of couples. We used speech data collected from 101 Dutch-speaking couples in Belgium who engaged in 10-minute long conversations in the lab. We extracted acoustic features from (1) the audio segments with the most extreme positive and negative ratings, and (2) the ending of the audio. We used transfer learning in which we extracted these acoustic features with a pre-trained convolutional neural network (YAMNet). We then used these features to train machine learning models - support vector machines - to predict the end-of-conversation valence ratings (positive vs negative) of each partner. The results of this work could inform how to best recognize the emotions of couples after conversation-sessions and eventually, lead to a better understanding of couples' relationships either in therapy or in everyday life.
引用
收藏
页码:17 / 21
页数:5
相关论文
共 45 条
[1]   Toward automating a human behavioral coding system for married couples' interactions using speech acoustic features [J].
Black, Matthew P. ;
Katsamanis, Athanasios ;
Baucom, Brian R. ;
Lee, Chi-Chun ;
Lammert, Adam C. ;
Christensen, Andrew ;
Georgiou, Panayiotis G. ;
Narayanan, Shrikanth S. .
SPEECH COMMUNICATION, 2013, 55 (01) :1-21
[2]   Towards Real-Time Multimodal Emotion Recognition among Couples [J].
Boateng, George .
PROCEEDINGS OF THE 2020 INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, ICMI 2020, 2020, :748-753
[3]  
Boateng George, 2020, 1 MOM EM EL CAPT WOR
[4]   MSP-IMPROV: An Acted Corpus of Dyadic Interactions to Study Emotion Perception [J].
Busso, Carlos ;
Parthasarathy, Srinivas ;
Burmania, Alec ;
AbdelWahab, Mohammed ;
Sadoughi, Najmeh ;
Provost, Emily Mower .
IEEE TRANSACTIONS ON AFFECTIVE COMPUTING, 2017, 8 (01) :67-80
[5]   IEMOCAP: interactive emotional dyadic motion capture database [J].
Busso, Carlos ;
Bulut, Murtaza ;
Lee, Chi-Chun ;
Kazemzadeh, Abe ;
Mower, Emily ;
Kim, Samuel ;
Chang, Jeannette N. ;
Lee, Sungbok ;
Narayanan, Shrikanth S. .
LANGUAGE RESOURCES AND EVALUATION, 2008, 42 (04) :335-359
[6]   EMOTIONAL BEHAVIOR IN LONG-TERM MARRIAGE [J].
CARSTENSEN, LL ;
GOTTMAN, JM ;
LEVENSON, RW .
PSYCHOLOGY AND AGING, 1995, 10 (01) :140-149
[7]   Predicting Behavior in Cancer-Afflicted Patient and Spouse Interactions using Speech and Language [J].
Chakravarthula, Sandeep Nallan ;
Li, Haoqi ;
Tseng, Shao-Yen ;
Reblin, Maija ;
Georgiou, Panayiotis .
INTERSPEECH 2019, 2019, :3073-3077
[8]   A Review and Meta-Analysis of Multimodal Affect Detection Systems [J].
D'Mello, Sidney K. ;
Kory, Jacqueline .
ACM COMPUTING SURVEYS, 2015, 47 (03)
[9]   Complex affect dynamics add limited information to the prediction of psychological well-being [J].
Dejonckheere, Egon ;
Mestdagh, Merijn ;
Houben, Marlies ;
Rutten, Isa ;
Sels, Laura ;
Kuppens, Peter ;
Tuerlinckx, Francis .
NATURE HUMAN BEHAVIOUR, 2019, 3 (05) :478-491
[10]   A Review of Generalizable Transfer Learning in Automatic Emotion Recognition [J].
Feng, Kexin ;
Chaspari, Theodora .
FRONTIERS IN COMPUTER SCIENCE, 2020, 2