Teaching Composition Quality at Scale Human Judgment in the Age of Autograders

We describe an effort to improve the composition quality of student programs: the property that a program can be understood effectively by another person. As a semester-long component of UC Berkeley's first course for majors, CS 61A, we gave students composition guidelines, scores, and qualitative feedback-all generated manually by a course staff of 10 graders for over 700 students. To facilitate this effort, we created a new online tool that allows instructors to provide feedback efficiently at scale. Our system differs from recently developed alternatives in that it is a branch of an industrial tool originally developed for internal code reviews at Google and used extensively by the open-source community. We found that many of the features designed for industrial applications are well-suited for instructional use as well. We extended the system with permissions controls and comment memories tailored for giving educational feedback. Using this tool improved the consistency of the feedback we gave to students, the efficiency of generating that feedback, and our ability to communicate that feedback to students. Emphasizing composition throughout the course improved the composition of our students' code. The quality of student programs improved by a statistically significant margin (p < 0.01) over those from a previous semester, measured by a blind comparison of student submissions.