Estimation of Programming Understanding by Time Series Analysis of Code Puzzles

In programming education, it is desirable for instructors to stand beside learners and monitor their answering process to assess the individual's actual ability. However, it seems to be impossible in large-group lectures at educational institutions or newcomer education at companies. Therefore, instructors attempt to grasp the understanding status of many learners at once by using written tests and e-learning to find out the learners who need instruction. They examine the learner's knowledge such as algorithm and syntax. However, in reality, not a few learners fail to acquire the skill of writing source codes. This kind of situation implies that the programming ability of learners cannot be measured only by knowledge tests or the data obtained from answer results. The purpose of this study is to estimate the understanding of programming, focusing on the thinking process. This paper analyzes a time series of operations of learners working on code puzzles, where they arrange code fragments. Since we assumed learners with low understanding are different from those with high in terms of the consistency of blocks of code fragments to be touched, we modeled it using a hidden Markov model. The proposed method estimates their perspectives on how fragments are built up to achieve given requirements. The results of an experiment have shown that the calculated hidden Markov model produces meaningful interpretable values. Furthermore, the values show significant indices that machine learning models can explain the understanding of learners.