The multimodal representation of emotion in film: Integrating cognitive and semiotic approaches

This study provides a semiotic theorization of how emotion is represented in film to complement the cognitive approach, which focuses on how film elicits emotion from viewers. Drawing upon social semiotic theories and cognitive theories of emotion, we develop a multimodal framework in which filmic representation of emotion is seen as combinations of semiotic choices derived from cognitive components of emotion. The semiotic model is employed to investigate how emotive meaning is realized through verbal and nonverbal resources. At the discursive level of film, the choices available in the shot organization of eliciting condition and expression are examined. The paper demonstrates how the social semiotic approach, combined with cognitive theory of emotion structure, is able to provide a comprehensive theoretical account of how various film techniques represent emotion. It is also significant for the study of viewer emotion, which to a large degree stems from character emotion.