Using machine learning to generate engaging behaviours in immersive virtual environments

Our work aims at implementing autonomous agents for Immersive Virtual Reality (IVR). With the advances in IVR environments, users can be more engaged and respond realistically to the events delivered in IVR, a state described in literature as presence. Agents with engaging verbal and nonverbal behaviour help preserve the sense of presence in IVR. For instance, gaze behaviour plays an important role, having monitoring and communicative functions. The initial step is to look at a machine learning model that generates flexible and contextual gaze behaviour and takes into account the rapport between the user and the agent. In this paper, we present our progress to date on the problem of creating realistic nonverbal behaviour. This includes analysing a multimodal dyad data, creating a data-processing pipeline, implementing a Hidden Markov Model and linking the Python scripts with the VR game engine (Unity3D). Future work consists of using richer data for more complex machine learning models, with a final aim of integrating the gaze model (plus future nonverbal behaviour models) into an autonomous virtual character framework.