Using Natural Language Processing and Machine Learning to Replace Human Content Coders

Content analysis is a common and flexible technique to quantify and make sense of qualitative data in psychological research. However, the practical implementation of content analysis is extremely labor-intensive and subject to human coder errors. Applying natural language processing (NLP) techniques can help address these limitations. We explain and illustrate these techniques to psychological researchers. For this purpose, we first present a study exploring the creation of psychometrically meaningful predictions of human content codes. Using an existing database of human content codes, we build an NLP algorithm to validly predict those codes, at generally acceptable standards. We then conduct a Monte-Carlo simulation to model how four dataset characteristics (i.e., sample size, unlabeled proportion of cases, classification base rate, and human coder reliability) influence content classification performance. The simulation indicated that the influence of sample size and unlabeled proportion on model classification performance tended to be curvilinear. In addition, base rate and human coder reliability had a strong effect on classification performance. Finally, using these results, we offer practical recommendations to psychologists on the necessary dataset characteristics to achieve valid prediction of content codes to guide researchers on the use of NLP models to replace human coders in content analysis research. Translational Abstract As psychological research enters the "Big Data" era where much richer, yet often unstructured data can be utilized to answer novel research questions, it becomes increasingly important for researchers and practitioners to possess necessary techniques to analyze these data. Content analysis of psychological data is an often-utilized technique with practical shortcomings where human coders process and code, by psychological science standards, large amounts of data. The present study demonstrated that NLP and ML can be engineered to predict human codes with sufficiently high accuracy to maintain psychometric rigor, thereby saving considerable time and human coder efforts. Furthermore, based on the results of a Monte-Carlo simulation, practical guidelines and recommendations on the necessary dataset characteristics (e.g., sample size) to achieve valid prediction of content codes were provided for practitioners who wish to adopt this technique.