Generation of Code Tracing Problems From Open-Source Code

When developing automatic quizzing systems and intelligent tutoring systems, significant effort has to be spent on developing the problem bank. Question and problem generation is a field of study concerning automating this routine work. The two most common methods of problem generation are constrained random generation and template-based generation, but each of them has disadvantages. In this work, we study a possibility of generating code-tracing learning problems from existing open-source code. We generated a set of learning problems and evaluated their distinctness from human-authored problems and readiness for usage in the learning process. Both teachers and students showed the rate of determining machine-generated problems only slightly above random guessing. Teachers strongly agreed with the problems' relevance and agreed with their suitability for the learning process. Automatic labeling to filter desired problems for the assignment includes used concepts, possible errors during solving, and difficulty estimates. The studied type of learning problem required little additional data to add to the code; our further work will concern problem types with more dynamic data to overcome this limitation.