Development of VAScoR: A rubric to qualify and score responses to the views of nature of science (VNOS) questionnaire

We report on the development of a rubric to reliably qualify and score responses to the Views of Nature of Science Questionnaire (VNOS): The VNOS Analysis and Scoring Rubric (VAScoR). The VAScoR is designed to (a) provide systematic guidance for the qualitative analysis, and score assignment to nuanced categories, of VNOS responses, (b) explicitly scaffold qualitative inferencing and standardize score assignment to substantially lessen the burden of, and variance in, analyzing and scoring the VNOS, and (c) improve the viability and meaningfulness of cross-study comparisons drawing on VNOS data. The rubric adopted the VNOS's consensus NOS framework and further delineated core and related elements across 10 target NOS aspects. The VAScoR's reliability was examined in two studies that drew on VNOS questionnaires completed by 185 preservice secondary science teachers (58% female; 126 undergraduate and 59 graduate students) enrolled over several years in a combined undergraduate and graduate licensure program in a large U.S. Midwestern university. In Study I, VAScoR analyses of 86 VNOS questionnaires undertaken by a single author were used to examine the rubric's intra-rater reliability, which resulted in a robust Cronbach's alpha value of 0.81. In Study II, analyses by four authors of a randomly generated, overlapping set of 18 questionnaires were used to examine inter-rater reliability, which was supported with substantial consensus among raters as indicated by a Cohen's kappa of 0.71. Further evidence for the VAScoR's inter-rater reliability was indicated by moderate to strong consistency among four raters with an overall Pearson's correlation coefficient of 0.82, and coefficient values ranging from 0.77 to 0.89 for six possible rater pairings.