Identification and classification of indicators for evaluating health information systems

Background: Several evaluation methods are used to determine the advantages and disadvantages of healthcare information systems and their contribution to attaining organizational goals. Despite the existence of many evaluation frameworks, there is no comprehensive set of indicators that evaluate different dimensions of information systems. This study aimed to develop a set of indicators for evaluating health information systems. Methods: This research was conducted in three phases. First, based on a literature review of PubMed, Web of Science, Scopus, and Embase databases, studies using the health information system evaluation methods were extracted. Second, consecutive focus group meetings were held with scientific and executive experts to discuss the list of evaluation indicators extracted from the studies. In these meetings, the experts agreed on including, removing, adding, combining, and grouping the indicators. Third, the indicators were weighted using the Analytical Network Process (ANP) method, and the set of evaluation indicators was finalized. Results: The review of 177 relevant articles resulted in the extraction of 360 indicators. During the focus group meetings, 174 overlapping and duplicate indicators were eliminated and 61 indicators were added to the model based on experts ' suggestions. The remaining 247 indicators were classified into a four-level hierarchy. The final set consisted of 4 dimensions, 16 criteria, 47 markers, and 180 indicators. Conclusion: We developed a comprehensive general set of indicators that helps researchers, designers, and developers of health information systems to evaluate different dimensions of these systems. This set can also be used to improve the design of relevant systems.