Concordance between facial micro-expressions and physiological signals under emotion elicitation

Various modalities have been leveraged for affective computing, alone or combined, such as facial expressions, speech intonations, peripheral physiological signals, and brain activities. Previous studies have shown that the multimodal fusion of affective data usually improves performance. However, the internal interactive mechanism among different modalities is rarely studied. In this paper, we investigated the concordance between facial micro-expressions and physiological signals under high arousal emotion elicitation with strict synchronization. By linking the onset of micro-expressions with physiological signals, a series of epoch durations were created to cover the potential reaction delay that may vary with different physiological signals. The experimental results show a significant correlation between the appearance of micro-expressions and time-domain features of heart rate variability, but not respiration or electrodermal activity related features. These findings indirectly verify the feasibility and reliability of micro-expression as a measure for non-contact genuine emotion recognition and would be beneficial for the fusion of micro-expression and physiological signals for more robust affective computing and their applications in public security and mental health. (c) 2022 Elsevier B.V. All rights reserved.