Evaluating Agreement: Conducting a Reliability Study

Instruments that are useful in clinical or research practice will, when the object of measurement is stable, yield similar results when applied at different times, in different situations, or by different users. Studies that measure the relation of differences between patients or subjects and measurement error (reliability studies) are becoming increasingly common in the orthopaedic literature. In this paper, we identify common aspects of reliability studies and suggest features that improve the reader's confidence in the results. One concept serves as the foundation for all further consideration: in order for a reliability study to be relevant, the patients, raters, and test administration in the study must be similar to the clinical or research context in which the instrument will be used. We introduce the statistical measures that readers will most commonly encounter in reliability studies, and we suggest an approach to sample-size estimation. Readers interested in critically appraising reliability studies or in developing their own reliability studies may find this review helpful.