RAIF: A deep learning-based architecture for multi-modal aesthetic biometric system

Human aesthetics play a significant role in video game development, emotional-aware robot design, online recommender systems, digital human, and other domains of research focusing on human-computer interactions. Social network user recognition based on aesthetic preferences is an emerging research domain. In this paper, a novel deep learning architecture is proposed for multi-modal audio-visual person identification that combines audio and visual aesthetic features. A pre-trained ResNet architecture is utilized to extract high-level features from a set of user-preferred audio and image samples. A novel deep learning-based fusion technique called residual-aided intermediate fusion (RAIF) is introduced in order to effectively merge the audio and visual features. The proposed RAIF method achieved an accuracy of 98% and a loss of 0.01 on a proprietary multi-modal dataset, indicating its effectiveness in fusing audio and visual information.