Application of the Delphi Method in Computer Science Principles Rubric Creation

Growing public demand for computer science (CS) education in K-12 schools requires an increase in well-qualified and well-supported computing teachers. To alleviate the lack of K-12 computing teachers, CS education researchers have focused on hosting professional development workshops to prepare in-service teachers from other disciplines to teach introductory level computing courses. In addition to the curriculum knowledge and pedagogical content knowledge taught in the professional development workshops, these new teachers need support in computer science subject matter knowledge throughout the school year. In particular, these new teachers find it difficult to grade programs and labs. This research study uses two variations of the Delphi Method to create learning-oriented rubrics for Computer Science Principles teachers using the Beauty and Joy of Computing curriculum. To perform this study we implemented (1) a heavy-weight, heterogeneous wide-net Delphi, and (2) a lower-weight, homogeneous Delphi composed of master teachers. These methods resulted in the creation of two systematically- and rigorously-created rubrics that produce consistent grading and very similar inter-rater reliabilities.