Lessons from evaluating an automated patient severity index

Objective: To report lessons learned from evaluation of an automated interface between a hospital clinical information system and a severity of illness index. Design: A system was developed to convert coded electronic patient findings from the HELP System at LDS Hospital into the attributes used by the Computerized Severity Index (CSI) to calculate a severity of illness score. Performance was assessed by comparing the automated CSI score with the manual CSI score (from paper chart review) and by evaluating changes introduced by augmenting the manual CSI score with verified patient data discovered by the automated CSI method. Measurements: The strengths and weaknesses of each method are presented. Results: The automated CSI score matched the manual CSI score in 61% of the cases. Sources of errors were analyzed. When the automated score was in error, two-thirds of the time it was due to the lack of codes in the HELP system representing CSI concepts; one-third of the time it was due to nurses not using established HELP system codes. Surprisingly, significant problems were also discovered in the manual system, making it difficult to define a ''gold standard.'' Conclusions: Automated computerized severity indices have great potential for future applicability once their performance exceeds that of the time-consuming manual chart review method. Neither automated nor manual methods are adequate at the present time. This area remains a fertile ground for future research.