Turning Words Into Numbers: Assessing Work Attitudes Using Natural Language Processing

Researchers and practitioners are often interested in assessing employee attitudes and work perceptions. Although such perceptions are typically measured using Likert surveys or some other closed-end numerical rating format, many organizations also have access to large amounts of qualitative employee data. For example, open-ended comments from employee surveys allow workers to provide rich and contextualized perspectives about work. Unfortunately, there are practical challenges when trying to understand employee perceptions from qualitative data. Given this, the present study investigated whether natural language processing (NLP) algorithms could be developed to automatically score employee comments according to important work attitudes and perceptions. Using a large sample of employees, algorithms were developed to translate text into scores that reflect what comments were about (theme scores) and how positively targeted constructs were described (valence scores) for 28 work constructs. The resulting algorithms and scores are labeled the Text-Based Attitude and Perception Scoring (TAPS) dictionaries, which are made publicly available and were built using a mix of count-based scoring and transformer neural networks. The psychometric properties of the TAPS scores were then investigated. Results showed that theme scores differentiated responses based on their likelihood to discuss specific constructs. Additionally, valence scores exhibited strong evidence of reliability and validity, particularly, when analyzed on text responses that were more relevant to the construct of interest. This suggests that researchers and practitioners should explicitly design text prompts to elicit construct-related information if they wish to accurately assess work attitudes and perceptions via NLP.