Reliability of credentialing examinations and the impact of scoring models and standard-setting policies

This article provides a brief history of developments in the assessment of reliability of credentialing examinations and presents some new results that highlight the complex interactions among examination scoring, standard setting, and the reliability and validity of pass-fail decisions. With respect to the second purpose, these interactions have become especially important recently because of the use of performance assessments in credentialing examinations. The main points of the article are that decision consistency is an important concept in evaluating credentialing examinations and that an estimate of the error in scores near the passing score or standard is essential. Also, the influence of scoring models and standard-setting policies on decision consistency and accuracy can be substantial. These factors need to be considered in arriving at final decisions about examination scoring and standard setting.