Quantitative Evaluation of Student Engagement in a Large-Scale Introduction to Programming Course using a Cloud-based Automatic Grading System

In this WIP Research to Practice paper, we explored the impact of integrating the university's learning management system and an automatic grading system in delivering a large-scale introduction to programming course in 2 consecutive quarters. Our initial approach utilizes an on-demand standalone automatic grading system and a separate assignment submission portal on Canvas. After evaluating our performance and specific student feedback, we integrated the assignment submission portal with the autograder system to provide a real-time objective assessment of assignments. The main improvement after enforcing assignment submission through the autograder (Stepik) was the noticeable improvement in the class average of assignment scores by 20.5% even though most of the test cases were hidden. Another interesting observation was the effect of our approach in decreasing the DFW rate to 12.5% from 46% and a considerable increase in the passing rate of female students, by 22%. We also noticed that in the second iteration of the course students who took the course as an elective were able to perform comparably and even better than students who took it as a requirement. It is also worth mentioning that using autograder helped students increase their code quality.