Remote Automated User Testing: First Steps toward a General-Purpose Tool

In this paper we explore options for conducting remote, unattended usability tests to enable users to participate in their own environments and time zones, including multiple users' participating at the same time in remote, unattended studies. We developed a general purpose tool, and code-named it "TCA" (for "Total Cost of Administration") to catalog and analyze database administrators' behavior within software. In this paper, we present example findings from the data collected over a period of 6 months. We analyzed users' deviations from the best paths through the software, in addition to collecting traditional measures such as time on task, error rate, and users perceptions, including satisfaction level. Further, we explore how this type of tool offers particular promise in benchmark studies, also collecting the ideal best-path performance that assumes error-free, expert user behavior, to compare to more real-world data collected initially in laboratory tests or over time in longitudinal field studies.