A method of scoring the severity of medication errors that does not require knowledge of patient outcomes was developed and tested. Thirty health care professionals from four U.K. hospitals scored 50 medication errors in terms of potential patient outcomes on a scale of 0 to 10, where 0 represented a case with no potential effect and 10 a case that would result in death. Sixteen error cases reported in the literature with actual patient outcomes were included among the cases to assess the validity of the scores. Ten of the errors were scored twice. The severity of the error cases, the occasion on which they were scored, the judge, each judge's profession, and the interactions between these were considered as potential sources of variability in scoring. The data were analyzed by applying generalizability theory to two models: one based on the 10 cases that were scored twice and ignoring the effect of differences in profession and one based on all 50 cases and ignoring the effect of the occasion of scoring. Generalizability coefficients for different numbers of judges and scoring occasions were calculated. A generalizability coefficient of 0.8 or more was considered to represent acceptable reliability. Most of the variance was attributable to differences in the cases. The analysis showed that to achieve a generalizability coefficient of more than 0.8, at least four judges would have to score each case, each on one occasion, with the mean score used as a severity indicator. A reliable, valid method of scoring the severity of medication errors that did not require knowledge of patient outcomes was developed; at least four judges were required in order to achieve reliable scores, and reliability was not affected by the professions of the judges or the number of occasions on which the errors were scored.
