A Field Study Comparing Approaches to Collecting Annotated Activity Data in Real-World Settings

Collecting ground-truth annotations for contextual data is vital to context-aware system development. However, current research lacks a systematic analysis of different approaches to collecting such data. We present a field experiment comparing three approaches: Participatory, Context-Triggered In Situ, and Context-Triggered Post Hoc, which involved users in recording and annotating activity data in real-world settings. We compared the quantity and quality of collected data using each approach, as well as the participant experience. We found Context-Triggered approaches produced more recordings, whereas the Participatory approach produced a greater amount of data with higher completeness and precision. Moreover, while participants appreciated automated recording and reminders for convenience, they highly valued having control over what and when to record and annotate. We conclude that user burden and user control are key aspects to consider when collecting and annotating contextual data with participants, and suggest features for a future tool focused on these two aspects.