Testing the usability of a personalized system: comparing the use of interviews, questionnaires and thinking-aloud

Personalized systems present each user with tailored content or output. Testing the usability of such a system must take some specific usability issues and the suitability of the personalized output into account. In this study, we evaluated a personalized search engine to compare the use of interviews, questionnaires and concurrent thinking-aloud to this avail. The interview and the questionnaire are the best methods to elicit comments on usability issues. Concurrent thinking-aloud turned out to be the best method to elicit comments on the perceived relevance of search results. When testing the usability of a personalized system it is wise to use a combination of concurrent thinking-aloud and the interview or the questionnaire.