Cognitive load classification of mixed reality human computer interaction tasks based on multimodal sensor signals

Evaluating cognitive load in mixed reality (MR) has become a significant challenge in human-computer interaction (HCI). To address this, we established an MR multimodal experimental platform with three distinct environments to induce varying levels of cognitive load. Participants engaged in MR-based CNC machine tool interaction tasks within these environments. Using the built-in sensors of the HoloLens 2 mixed reality head-mounted display (MR-HMD) and wearable heart rate sensors, we collected device and physiological data from participants wearing the MR-HMD while performing these tasks. The cognitive load of participants was assessed by using the NASA-TLX questionnaire. Experimental results indicated that the operation time required in the MR environment increased by 49% under high cognitive load compared to low-load conditions. High-load environments also led to increased anxiety, frustration, and decreased performance among participants. Through comparative experiments, we identified suitable sensor data streams and algorithms for cognitive load classification and designed an MR digital twin factory cognitive load warning prototype system. This system utilizes an improved Transformer-CL algorithm, achieving a cognitive load classification accuracy of 95.83%. The system provides high cognitive load warnings, reducing the risks associated with high cognitive load tasks in MR work environments. © The Author(s) 2025.