Multimodal Search and Visualisation of Movies Based on Emotions Along Time in As Movies Go By

Largely due to the significant emotional impact they have on viewers and in their lives, movies are a powerful vehicle for culture and education and one of the most important and impactful forms of entertainment. By making a huge amount of movies more accessible in pervasive services and devices, and helping in emotion recognition and classification, technology has been playing an important role, and it is becoming more pertinent the ability to search, visualize and access movies based on their emotional impact, although emotions are seldom taken into account in these systems. In this paper, we characterize the challenges and approaches in this scenario, then present and evaluate, at two different stages, interactive means to visualize and search movies based on their dominant and actual emotional impact along the movie, with different and personalizable models and modalities, in As Movies Go By. In particular through emotional highlights in words, colors, emojis and trajectories, by drawing emotional blueprints or through users’ emotional states, with the ability to get us into a movie in serendipitous moments. © The Author(s) 2025.