Automatic Coding Mechanisms for Open-Ended Questions in Journalism Surveys: An Application Guide

Answers from open-ended questions are a valuable part of journalism surveys. However, due to the expense and difficulties involved in manual coding, the current situation, in which open-ended questions are used and analyzed in large-scale web surveys, is not satisfactory. This article reviews the types, coding tasks, and automatic coding techniques of open-ended questions. We propose a five-step procedure on how to analyze open-ended questions through an automatic coding approach, the process of which is as follows: (a) locate the type of open-ended question, (b) choose the corresponding coding task, (c) adopt the appropriate automatic coding techniques, (d) perform the analysis, and (e) evaluate and interpret the results. We demonstrate this with survey data from Reuters Digital News Reports of 2019. Our proposed framework can serve as a practical guide for analyzing open-ended questions with automatic coding and also promote open science in journalism research. We conclude that although automatic coding cannot entirely replace human coding, the constant refinement of the statistical models and the promotion of open sharing of textual data will gradually render autocoding a standard tool for researchers in journalism and communication.