Emotion recognition based on brain-like multimodal hierarchical perception

Emotion recognition has gained prominence in diverse applications ranging from safe driving and e-commerce to healthcare. Traditional approaches have often relied on single-modal information such as visual, audio, or text, resulting in limitations in both reliability and robustness. To address these shortcomings, we introduce a brain-inspired computing model for emotion recognition that mimics the hierarchical processing characteristics of human cognitive functions. This innovative model accommodates multimodal information cohesively, aiming to emulate the human cognitive process across visual, audio, and text. To gain a better grasp of our brain-like hierarchical perception architecture, we stratify the model into three key layers: feature extraction, fusion, and decision-making. This structure integrates cognitive mechanisms with machine learning algorithms for enhanced performance. Specifically, we begin by extracting deep features that emulate the human brain’s perception of emotional cues. These features are then synthesized using a cross-attention mechanism to explore inter-modal correlations. Finally, the aggregated emotional data is categorized and recognized. Experimental results indicate that our approach achieves an average recognition accuracy of 82.7% across four distinct emotion classifications, showcasing its effectiveness and offering a fresh perspective for multimodal emotion recognition. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2023.