Investigating mental workload across task modalities: a multimodal analysis using pupillometry

Understanding mental workload is challenging due to its multidimensional nature and varying sensitivities of its primary measures: task performance, perceived workload, and physiological responses. This study investigates the effects of task load on performance, perceived workload, and pupil measures across different information modalities. A within-subjects experiment involving three tasks (digit span, matrix span, and dual n-back) was conducted with three task load levels. Workload measures included accuracy/sensitivity, reaction time, NASA-TLX, peak pupil diameter, and peak pupil latency. Consistent patterns of associations between task load and these workload measures were found across the three modalities. Workload measures revealed a nuanced interplay between task modality and task load. Robustness of peak pupil latency, accuracy, and NASA-TLX was highlighted across verbal and visual modalities, while peak pupil diameter showed a weaker impact with differences between modalities. The findings encourage multivariate assessment of mental workload to account for different task modalities. Interacting with current human-machine interfaces requires operators to comprehend visual and verbal information. Therefore, a multivariate approach accounting for different types of information as well as different methods of workload assessment is essential for developing a more comprehensive model of mental workload.