Investigating human reading behavior during sentiment judgment

Sentiment analysis is an essential task in natural language processing researches. Although existing works have gained much success with both statistical and neural-based solutions, little is known about the human decision process while performing this kind of complex cognitive task. Considering recent advances in human-inspired model design for NLP tasks, it is necessary to investigate the human reading and judging behavior in sentiment classification and adopt these findings to reconsider the sentiment analysis problem. In this paper, we carefully design a lab-based user study in which users' fine-grained reading behaviors during microblog sentiment classification are recorded with an eye-track device. Through systematic analysis of the collected data, we look into the differences between human and machine attention distributions and the differences in human attention while performing different tasks. We find that (1) sentiment judgment is more like an auxiliary task of content comprehension for humans. (2) people have different reading behavior patterns while reading microblog posts with varying labels of sentiment. Based on these findings, we build a human behavior-inspired sentiment prediction model for microblog posts. Experiment results on public-available benchmarks show that the proposed classification model outperforms existing solutions over 2.13% in terms of macro F1-score by introducing behavior features. Our findings may bring insight into the research of designing more effective and explainable sentiment analysis methods.