An engagement-aware predictive model to evaluate problem-solving performance from the study of adult skills' (PIAAC 2012) process data

被引:2
作者
Shin, Jinnie [1 ]
Wang, Bowen [1 ]
Pinto Junior, Wallace N. [1 ]
Gierl, Mark J. [2 ]
机构
[1] Univ Florida, Coll Educ, Res & Evaluat Methodol, Gainesville, FL 32611 USA
[2] Univ Alberta, Dept Educ Psychol, Edmonton, AB, Canada
关键词
PIAAC; PS-TRE; Long-short-term-memory (LSTM); Attention; Engagement; Problem-solving performance; Log information; COMPUTATIONAL PSYCHOMETRICS; RESPONSE-TIME; MOTIVATION; NETWORKS; PATTERNS; BEHAVIOR; TASK;
D O I
10.1186/s40536-024-00194-y
中图分类号
G40 [教育学];
学科分类号
040101 ; 120403 ;
摘要
The benefits of incorporating process information in a large-scale assessment with the complex micro-level evidence from the examinees (i.e., process log data) are well documented in the research across large-scale assessments and learning analytics. This study introduces a deep-learning-based approach to predictive modeling of the examinee's performance in sequential, interactive problem-solving tasks from a large-scale assessment of adults' educational competencies. The current methods disambiguate problem-solving behaviors using network analysis to inform the examinee's performance in a series of problem-solving tasks. The unique contribution of this framework lies in the introduction of an "effort-aware" system. The system considers the information regarding the examinee's task-engagement level to accurately predict their task performance. The study demonstrates the potential to introduce a high-performing deep learning model to learning analytics and examinee performance modeling in a large-scale problem-solving task environment collected from the OECD Programme for the International Assessment of Adult Competencies (PIAAC 2012) test in multiple countries, including the United States, South Korea, and the United Kingdom. Our findings indicated a close relationship between the examinee's engagement level and their problem-solving skills as well as the importance of modeling them together to have a better measure of students' problem-solving performance.
引用
收藏
页数:22
相关论文
共 65 条
[1]  
Ai F., 2019, Concept-aware deep knowledge tracing and exercise recommendation in an online learning system
[2]  
[Anonymous], 2012, Literacy, Numeracy and Problem Solving in Technology-rich Environments: Framework for the OECD Survey of Adult skills
[3]  
Barber W, 2015, ELECTRON J E-LEARN, V13, P59
[4]  
Braun H, 2011, TEACH COLL REC, V113, P2309
[5]   Semantic Aware Attention Based Deep Object Co-segmentation [J].
Chen, Hong ;
Huang, Yifei ;
Nakayama, Hideki .
COMPUTER VISION - ACCV 2018, PT IV, 2019, 11364 :435-450
[6]   XGBoost: A Scalable Tree Boosting System [J].
Chen, Tianqi ;
Guestrin, Carlos .
KDD'16: PROCEEDINGS OF THE 22ND ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING, 2016, :785-794
[7]   Model-Based Treatment of Rapid Guessing [J].
Deribo, Tobias ;
Kroehne, Ulf ;
Goldhammer, Frank .
JOURNAL OF EDUCATIONAL MEASUREMENT, 2021, 58 (02) :281-303
[8]  
Eseryel D, 2014, EDUC TECHNOL SOC, V17, P42
[9]  
Goldhammer F., 2016, OECD Education Working Papers, V133, P3, DOI DOI 10.1787/5JLZFL6FHXS2-EN
[10]   The Time on Task Effect in Reading and Problem Solving Is Moderated by Task Difficulty and Skill: Insights From a Computer-Based Large-Scale Assessment [J].
Goldhammer, Frank ;
Naumann, Johannes ;
Stelter, Annette ;
Toth, Krisztina ;
Roelke, Heiko ;
Klieme, Eckhard .
JOURNAL OF EDUCATIONAL PSYCHOLOGY, 2014, 106 (03) :608-626