Multi-party focus of attention recognition in meetings from head pose and multimodal contextual cues

This paper presents investigations on visual focus of attention (VFOA) recognition in meetings from audio-visual perceptual cues. Rather than independently recognizing the VFOA of each participant from his own head pose, we propose to recognize participants' VFOA jointly in order to introduce context dependent interaction models that relates to group activity and the social dynamics of communication. To this end, we designed an input-output hidden Markov model (IOHMM), whose hidden states are the joint VFOA of all participants, and whose main observations are the head poses. Interaction models are introduced in the form of contextual cues that affect the temporal evolution of the joint VFOA sequence, allowing us to model group dynamics that accounts for people's tendency to share the same focus, or to have their VFOA driven by contextual cues such as slide activity or the participant speaking activity. The model is rigorously evaluated on a publicly available dataset of 4 real meetings of 23min on average, showing an overall 10% relative performance increase w.r.t. the independent recognition case.