Taking Stock of Concept Inventories in Computing Education: A Systematic Literature Review

Background and context. Concept inventories (CIs) are a widely used tool in STEM education that can help instructors identify specific misconceptions students hold about key concepts. Over the past several years, much research has been published contributing to CIs in computer science education. Objectives. This research aims to investigate the state of CIs in computer science education through a systematic literature review built around two research questions: 1) What do we know about CIs for computer science today, and 2) Have researchers been able to effectively tackle the 4 challenges specific to computer science CIs outlined by Taylor et al. in their previous 2014 literature review on this topic? Method. We conducted a systematic literature review focusing on CIs in computer science education. We included research from any country, provided it was published in English between 2004 and 2022. We searched with key terms across four databases (Compendex, Inspec, Education Source, and ERIC) and collated a final list of 65 studies for analysis from 175 initial results. Findings. From these 65 papers, we identified 33 total CIs in computer science, 12 of which are currently validated. Of the 65 studies, the majority used established techniques for choosing concepts, identifying misconceptions, and/or developing questions, but a few (n = 4) reported on successful alternative techniques. We found that although no papers explicitly articulated an objective to tackle the aforementioned challenges, they are a recurring theme in the literature and have been addressed in various ways. Implications. At the time of the previous literature review, there were few computer science CIs. Our findings are promising, suggesting that researchers have found effective ways to tackle Taylor et al.'s challenges and that the number of CIs across computer science is gradually increasing. Future work is warranted regarding non-traditional techniques researchers proposed for developing CIs.