Convenience Sampling Methods in Psychology: A Comparison Between Crowdsourced and Student Samples

Traditional convenience sampling methods relying solely on student participants have raised concerns about the external validity of psychology research findings. Crowdsourced participant sampling methods, like Qualtrics online panels, may offer a viable alternative by minimizing barriers to external validity. However, our understanding of the reliability and external validity of data from these sources is limited. Utilizing secondary data from a larger study conducted in Canada, the present study sought to explore differences between a Qualtrics-recruited sample (n = 295) and a sample recruited via a university participant pool (n = 270) on sociodemographic characteristics, participant response characteristics, and mental health variables. Differences were found between sample source on sociodemographic characteristics such as participant age, racial background, marital status, and occupational status. The samples differed on participant response characteristics, with the Qualtrics sample demonstrating a higher failure rate on the total attention check measure and shorter survey completion time compared to the student sample. The sample sources also differed on mental health, with a higher prevalence of self-reported mental disorder diagnosis and higher levels of depression, anxiety, stress, and social anxiety symptoms in the Qualtrics sample compared to the student sample. Findings suggest that although Qualtrics panel data may be of lower quality than that of student pool samples with regard to attentive responding, it may still provide a useful recruitment source for clinical and occupational psychology researchers. When deciding on a recruitment method, researchers in psychology should aim to balance convenient access to representative samples with data quality.