Detecting Learning Stages within a Sensor-Based Mixed Reality Learning Environment Using Deep Learning

被引:3
|
作者
Ogunseiju, Omobolanle [1 ]
Akinniyi, Abiola [2 ]
Gonsalves, Nihar [2 ]
Khalid, Mohammad [2 ]
Akanmu, Abiola [3 ]
机构
[1] Georgia Tech, Coll Design, Sch Bldg Construct, Atlanta, GA 30332 USA
[2] Virginia Tech, Myers Lawson Sch Construct, Blacksburg, VA 24060 USA
[3] Virginia Tech, Myers Lawson Sch Construct, Construct Engn & Management, Blacksburg, VA 24060 USA
基金
美国国家科学基金会;
关键词
Mixed reality; Laser scanning; Deep learning; Eye tracking; Construction education; EYE-TRACKING; VIRTUAL-REALITY; CLASSIFICATION; FIXATION; FRAME;
D O I
10.1061/JCCEE5.CPENG-5169
中图分类号
TP39 [计算机的应用];
学科分类号
081203 ; 0835 ;
摘要
Mixed reality has been envisioned as an interactive and engaging pedagogical tool for providing experiential learning experiences and potentially enhancing the acquisition of technical competencies in construction engineering education. However, to achieve seamless learning interactions and automated learning assessments, it is pertinent that the mixed reality environments are intelligent, proactive, and adaptive to students' learning needs. With the potentials of artificial intelligence for promoting interactive, assistive, and self-reliant learning environments and the professed effectiveness of deep learning in other domains, this study explores an approach to developing a smart mixed reality environment for technical skills acquisition in construction engineering education. The study is based on the usability assessment of a previously developed mixed reality environment for learning sensing technologies such as laser scanners in the construction industry. In this study, long short-term memory (LSTM) and a hybrid LSTM and convolutional neural networks (CNN) models were trained with augmented eye-tracking data to predict students' learning interaction difficulties, cognitive development, and experience levels. This was achieved by using predefined labels obtained from think-aloud protocols and demographic questionnaires during laser scanning activities within a mixed reality learning environment. The proposed models performed well in recognizing interaction difficulties, experienced levels, and cognitive development with F1 scores of 95.95%, 98.52%, and 99.49% respectively. The hybrid CNN-LSTM models demonstrated improved performance with an accuracy of at least 20% higher than the LSTM models but at a higher inference time. The efficacy of the models for detecting the required classes, and the potentials of the adopted data augmentation techniques for eye-tracking data were further reported. However, as the model performance increased with data size, the computational cost also increased. This study sets precedence for exploring the applications of artificial intelligence for mixed reality learning environments in construction engineering education.
引用
收藏
页数:17
相关论文
共 50 条
  • [1] Enhancing Sensor-Based Mobile User Authentication in a Complex Environment by Deep Learning
    Weng, Zhengqiu
    Wu, Shuying
    Wang, Qiang
    Zhu, Tiantian
    MATHEMATICS, 2023, 11 (17)
  • [2] Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning
    Hammad, Issam
    El-Sankary, Kamal
    SENSORS, 2019, 19 (16)
  • [3] An Intelligent and Optimal Deep-Learning Approach in Sensor-Based Networks for Detecting Microbes
    Remya, S.
    Anjali, T.
    IEEE SENSORS JOURNAL, 2023, 23 (17) : 19367 - 19381
  • [4] Sensor-based Activity Recognition using Deep Learning: A Comparative Study
    Trabelsi, Imen
    Francoise, Jules
    Bellik, Yacine
    PROCEEDINGS OF 2022 8TH INTERNATIONAL CONFERENCE ON MOVEMENT AND COMPUTING, MOCO 2022, 2022,
  • [5] Deep learning for sensor-based activity recognition: A survey
    Wang, Jindong
    Chen, Yiqiang
    Hao, Shuji
    Peng, Xiaohui
    Hu, Lisha
    PATTERN RECOGNITION LETTERS, 2019, 119 : 3 - 11
  • [6] [LISA] Learning Analytics for Sensor-Based Adaptive Learning
    Fortenbacher, Albrecht
    Pinkwart, Niels
    Yun, Haeseon
    SEVENTH INTERNATIONAL LEARNING ANALYTICS & KNOWLEDGE CONFERENCE (LAK'17), 2017, : 592 - 593
  • [7] SENSOR-BASED CYBERATTACK DETECTION IN CRITICAL INFRASTRUCTURES USING DEEP LEARNING ALGORITHMS
    Yilmaz, Murat
    Catak, Ferhat Ozgur
    Gul, Ensar
    COMPUTER SCIENCE-AGH, 2019, 20 (02): : 213 - 243
  • [8] Deep Learning for Sensor-Based Rehabilitation Exercise Recognition and Evaluation
    Zhu, Zheng-An
    Lu, Yun-Chung
    You, Chih-Hsiang
    Chiang, Chen-Kuo
    SENSORS, 2019, 19 (04)
  • [9] Mixed reality teaching and learning environment
    Callaghan, Victor
    Gardner, Michael
    Horan, Bernard
    Scott, John
    Shen, Liping
    Wang, Minjuan
    HYBRID LEARNING AND EDUCATION, PROCEEDINGS, 2008, 5169 : 54 - 65
  • [10] A Mixed-reality Learning Environment
    Quint, Fabian
    Sebastian, Katharina
    Gorecky, Dominic
    2015 INTERNATIONAL CONFERENCE VIRTUAL AND AUGMENTED REALITY IN EDUCATION, 2015, 75 : 43 - 48