On Flexible Multiple Choice Questions With Parameters

Flexible multiple choice questions with configurable random answer options are a common feature in advanced e-assessment systems. Variable parameters can be used to create different versions of an exercise, each time a student looks at it. The approach can be applied to parameterize complete choices of an item as well as particular values appearing in the question and individual answers. However, plain uncontrolled randomization may result in a loss of exercise quality. In fact, writing good multiple choice questions is an established subject of research in classical test theory and item response theory, both subfields of psychometrics. In particular plausible distracters play an important role for measuring the learning outcome of students. A distracter is considered as plausible, when it is based on a common misconception about the task. In this context the quality of a distracter can be measured by evaluating how often the distracter was selected. Distracters that are selected too infrequently (e.g. < 5%) are called non-functioning distracters. They are regarded to be ineffective and they should be removed from the item. On the other hand, one can assign a difficulty to a distracter, depending on its selection frequency. The more often a functioning distracter is selected, the more difficult it appears to be. This paper presents how we can use the concept of parameterizing multiple choice questions inside the e-assessment system JACK to design exercises with good distracters. In particular it is our aim to obtain a pool of functioning distracters with different levels of difficulty. This enables us to parameterize the distracters of an exercise depending on the context in which the exercise is used. We give a case study of a distracter analysis for an exercise that was used in a preparation course for mathematics at our university. We show how we can identify and remove non-functioning distracters from the exercise. We can group the remaining distracters by their level of difficulty and achieve promising progress towards adaptivity through this approach.