Using on-line surveys to measure three key constructs of the quality of human-computer interaction in web sites: psychometric properties and implications

On-line surveys are now an important tool for data collection on the World Wide Web (the Web). Determining the psychometric properties of key constructs such as disorientation, ease of use and flow is of paramount importance in establishing the quality of users' interactions with web sites. The current study used techniques of experimental research and on-line surveys to investigate the psychometric properties of existing instruments for measuring these constructs using two response formats: visual analogue scale and Likert scale. A 2 x 2 design with response format and orientation support as independent variables was used. Ninety participants carried out an information retrieval task using an experimental on-line shopping site before completing the scales. Factor analysis confirmed the existence of three distinct scales that possessed high reliability. Evidence for validity, and to a lesser extent, sensitivity, was found. Although psychometric results generally converged, some differences between the two response formats were found. A framework for the comprehensive investigation of response formats of on-line questionnaires is proposed as a basis for future research. Practical implications for the on-line measurement of the quality of users' interactions with web-based systems are discussed. (C) 2003 Elsevier Science Ltd. All rights reserved.