Enhancing Speech Emotion Recognition for Real-World Applications via ASR Integration

Speech Emotion Recognition (SER) is a major focus of affective computing and has developed rapidly over the past few decades. However, the lack of large-scale emotional speech data has hindered the application of SER. As a result, most SER research conducted on existing corpora in the lab is not applicable to the wild. To address this issue, we propose integrating Automatic Speech Recognition (ASR), which is widely used in the wild, into SER for real-world applications. Specifically, we aim to investigate the mutual impact of speech and emotion recognition to understand how ASR performs on emotional speech and leverage the representations and transcripts from ASR models to develop a robust SER model. This research is expected to alleviate the data scarcity problem in SER and enable its use in various real-world applications. Furthermore, our findings can expand the usage of ASR in complicated speech scenarios (e.g., emotional speech) and advance other speech tasks (e.g., recognizing affective and health states) that face similar issues as SER.