Identification of Careless Responding in Ecological Momentary Assessment Research: From Posthoc Analyses to Real-Time Data Monitoring

With the emerging ubiquity of cell phones, ecological momentary assessment (EMA) as a set of methods enable researchers to study momentary social, psychological, and affective responses to everyday life. Additionally, EMA enables researchers to acquire longitudinal data without the need for multiple lab visits. As the use of EMA in research increases, so too does the necessity of determining what constitutes valid or careless individual EMA responses to ensure validity and replicability of findings. Currently, EMA studies solely consider the response rate of a participant for exclusion. Yet, other features of an assessment can help to determine whether a response is careless or implausible. Here, we examined over 18,000 EMA text message responses of individual affect items to derive a data-driven model of what constitutes a "careless response." Results from this study indicate that an overly fast time to complete items (<= 1 s), an overly narrow within assessment response variance (SD <= 5), and the percentage of items that fall at the mode (>= 60%) are independent and reliable indicators of a careless response. Excluding careless responses such as these remove implausible positive correlations among psychometric antonyms (e.g., relaxed and anxious). Further, by identifying and removing careless responses, we also identify careless responders, participants who could be removed from group analyses. We use these results to develop and introduce an R package, EMAeval, so EMA researchers may similarly identify careless responses and responders either online during data collection or posthoc, after data collection has completed.