STATISTICAL IMPLICATIONS OF THE CHOICE BETWEEN A DICHOTOMOUS OR CONTINUOUS TRAIT IN STUDIES OF INTEROBSERVER AGREEMENT

A frequently occurring question in the planning of studies is whether to measure the trait of interest on a continuous or dichotomous scale. An issue potentially relevant to this question centers around the resulting reliability of the measure. Assuming the continuous trait has a normal distribution, we quantify the effect of this decision on the overall number of subjects required to test whether a population reliability coefficient equals a specified criterion value. Two distinct cases are explored: the choice between a continuous response and a truly dichotomous response, and the case in which an inherently continuous response is dichotomized. In both cases, and particularly in the second, the loss of power associated with measuring the trait on a dichotomous scale is shown to be always severe and frequently prohibitive.