A Descriptive Analysis of Research on IT/IS Evaluation: A 25 Years Review

Research into Information Technology/Information Systems (IT/IS) evaluation is often considered in its infancy. However, through a quantitative review, the author argues that it is inappropriate to make such statement nowadays. Acknowledging the important role evaluation plays in the success of an IT/IS project, extensive efforts were made to develop effective evaluation approaches and the addressing of important issues when carrying out an evaluation, particularly in the last two decades. Questions like: What are the important issues in IT/IS evaluation research? Where do we put our efforts in? And what will be the direction for the future research? remain unanswered. This study presents a descriptive analysis of research on IT/IS evaluation over the last 25 five years, from 1986 to 2010, in five leading IS journals, including European Journal of Information Systems, Information Systems Research, Journal of Information Technology, MIS Quarterly and Journal of Management Information Systems. In total, 176 papers related to IT/IS evaluation are identified and reviewed in this study. Based on the Context, Content and Process model, IT/IS evaluation can be broken down to five interrelated elements: WHY evaluation is carried out, WHAT is evaluated, WHEN evaluation takes place, HOW evaluation is done and WHO is involved in evaluation. These elements in each paper reviewed are then identified, analysed and classified. Based on this, a previous classification of evaluation approaches is extended and criticized. The author suggests that evaluation can be classified into three streams: efficiency-driven, effectiveness-driven and understanding-driven. Details of each evaluation stream are described, and the relationship among three streams will also be discussed. Meanwhile, existing gaps in current research will be identified and suggestions are made to researchers interested in the topic for their future research.