Generalizability and decision studies: An example using conversational language samples

Observational data collected in ecologically valid measurement contexts are likely to be influenced by contextual factors irrelevant to the research question. Using multiple sessions and raters often improves the stability of scores for variables from such contexts. Generalizability (G) theory can be used to give researchers important information about sources and amounts of error in their observed scores. G theory also can be used to estimate the number of raters and sessions necessary to obtain reliable scores by conducting a decision (D) study. A general overview of G theory is provided in the present paper and its potential application for one type of a two-dimensional, fully crossed observational design is illustrated, using conversational language samples obtained from preschoolers with grammatical and phonological impairments. The paper includes the Internet link to an Excel (TM) spreadsheet, which calculates values necessary for the type of G and D studies exemplified here.