Studying Human Factors Aspects of Text Classification Task Using Eye Tracking

Text classification has a wide range of applications in today's world including filtering spam emails, identifying health conditions, categorizing news articles, business intelligence, and finding relevant legal documents. This has become scalable due to the use of supervised machine learning models which are usually trained on manually labelled text data and their performance is heavily dependent on the quality of training data. Manual text classification tasks involve a person reading the text and assigning the most appropriate category, which can incur a significant amount of cognitive load. Therefore, an in-depth understanding of human factors aspects of the text classification task is important, and it can help in determining the expected level of accuracy of human-labelled text as well as identifying the challenging aspects of the task. To the best of our knowledge, previous studies have not studied the text classification task from a human computer interaction (HCI) and human factors perspective. Our study is an early effort towards studying text classification task using eye-tracking information captured during the manual labelling process. We aim to analyze ocular parameters to understand the manual text classification process from an HCI perspective. We designed an eye-tracking study that involved 30 human subjects reading narratives of injury-related texts and selecting the best-suited category for the cause of injury events. Ocular parameters such as fixation count, average fixation duration, and pupil dilation values were recorded for each participant. Preliminary results from our study indicate that (a) reasonable level of average classification accuracy (75%) was observed for study participants, (b) a positive correlation between fixation count and fixation duration, and fixation count and pupil diameter was observed, and (c) we did not observe a consistent pattern between ocular parameters representative of cognitive load, the time taken to complete the task, and the classification accuracy, maybe due to underlying variations among humans and interpretability of textual narratives.