Utilising Learnersourcing to Inform Design Loop Adaptivity

Design-loop adaptivity refers to data-driven decisions that inform the design of learning materials to improve learning for student populations within adaptive educational systems (AES). Commonly in AESs, decisions on the quality of learning material are based on students' performance, i.e., whether engaging with the material led to learning gains. This paper investigates an alternative approach for design adaptivity, which utilises students' subjective ratings and comments to infer the quality of the learning material. This approach is in line with the recent shift towards learner-centred learning and learnersourcing, that aim to transform the role of students from passive recipients of content to active participants that engage with various higher-order learning tasks including evaluating the quality of resources. In this paper, we present a suite of aggregation-based and reliability-based methods that can be used to infer the quality of learning material based on student ratings and comments. We investigate the feasibility and accuracy of the methods in a live learnersourcing educational platform called RiPPLE that provides the capacity to capture subjective ratings and comments from students. Empirical data from the use of RiPPLE in a first-year course on information systems are used to evaluate the presented methods. Results indicate that the use of a combination of reliability-based methods provides an acceptable level of accuracy in determining the quality of learning resources.