Evaluating the agreement of users with usability problems identified by heuristic evaluation

Objective: Heuristic evaluation (HE) is an inspection method that can be used for identifying usability problems of health information systems. There is insufficient evidence to what extent these problems are experienced by users in their working environment. The objective of this study was to investigate the agreement of real end users of a health information system with the problems identified by HE. Methods: First, the emergency information system of educational hospitals in Kerman, Iran was evaluated by a heuristic evaluation. Then, a valid and reliable (Cronbach's alpha = 0.95) electronic questionnaire was designed based on the results of the first phase. Results: Among 127 identified problems, the highest number of problems was related to the "consistency and standards" heuristic (18.9%). The user agreement coefficient with heuristic results was 0.5. The highest agreement of the participants was on the problems related to the "help user recognize, diagnose and recover from errors" heuristic (78%). There was a significant relationship between participants age and work experience with a computer (p < 0.05). Conclusion: The agreement rate of real end users with the results of HE was at a medium level. Therefore, in some areas, the perception of evaluators in using this method is not consistent with the users experience with a system. This can result in identifying minor issues and ignoring important system-critical usability issues. The HE is appropriate for identifying specific problems, such as problems related to identifying and correcting errors, but in some areas, such as consistency and standards in the design of a system, it identifies minor or ineffective problems.