In Vivo or in Vitro? Influence of the Study Design on Crowdsourced Video QoE

Evaluating the QoE of video streaming and its influence factors has become paramount for streaming providers, as they want to maintain high satisfaction for their customers. In this context, crowdsourced user studies became a valuable tool to evaluate different factors which can affect the perceived user experience on a large scale. In general, we observed that most of these crowdsourcing studies either use an in vivo or an in vitro design. In vivo design means that the study participant has to rate the QoE of a video that is embedded in an application similar to a real streaming service, e.g., YouTube or Netflix. In vitro design refers to a setting, in which the video stream is separated from a specific service and thus, the video plays on a plain background. Although these designs vary widely, the results are often compared and generalized. Therefore, in this work, we investigate the influence of these two study design alternatives on the perceived QoE. In crowdsourced user studies, participants rate the video streaming with respect to different stalling patterns (no stalling, different positions) and study designs (in vivo or in vitro). Contrary to our expectations, the results indicate that there is statistically no significant influence of the study design on the perceived video QoE and acceptance. In addition, we found that the in vivo design does not reduce the test takers' attentiveness.