Exploring the reliability of generic and content-specific instructional aspects in physical education lessons

Heightened accountability pressures and an increased emphasis on teaching quality have directed scholarly attention to scrutinizing instruction, particularly with respect to issues of validity and reliability. However, these attempts have largely been directed toward core content areas and investigated generic or content-specific instructional aspects separately. Focusing on a less explored area, physical education, and concurrently attending to both instructional aspects, in this exploratory study, we examined whether the optimal lesson-rater combination needed to obtain reliable teaching-quality estimates differs depending on the type of instructional aspects considered. Data analysis of 147 lessons using generalizability theory suggested that either a 3-lesson-2-rater or a 4-lesson-1-rater combination yields sufficiently reliable estimates for nearly all dimensions examined (generic or content specific). Quality of student practice, however, required 11 lessons scored by 2 raters. These findings underline the importance of examining individual dimensions within a given observational instrument and the merit of carefully selecting and training raters.