Toward Automated Classroom Observation: Multimodal Machine Learning to Estimate CLASS Positive Climate and Negative Climate

被引:18
作者
Ramakrishnan, Anand [1 ]
Zylich, Brian [1 ]
Ottmar, Erin [1 ]
LoCasale-Crouch, Jennifer [2 ]
Whitehill, Jacob [1 ]
机构
[1] Worcester Polytech Inst, Worcester, MA 01609 USA
[2] Univ Virginia, Charlottesville, VA 22807 USA
基金
美国国家科学基金会;
关键词
Videos; Meteorology; Encoding; Machine learning; Activity recognition; Computer vision; Computer architecture; Automatic classroom observation; classroom assessment scoring system; facial expression recognition; auditory analysis; CHILD-CARE; ENGAGEMENT; QUALITY;
D O I
10.1109/TAFFC.2021.3059209
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
In this article we present a multi-modal machine learning-based system, which we call ACORN, to analyze videos of school classrooms for the Positive Climate (PC) and Negative Climate (NC) dimensions of the CLASS [1] observation protocol that is widely used in educational research. ACORN uses convolutional neural networks to analyze spectral audio features, the faces of teachers and students, and the pixels of each image frame, and then integrates this information over time using Temporal Convolutional Networks. The audiovisual ACORN's PC and NC predictions have Pearson correlations of 0.55 and 0.63 with ground-truth scores provided by expert CLASS coders on the UVA Toddler dataset (cross-validation on n 1/4 300 15-min video segments), and a purely auditory ACORN predicts PC and NC with correlations of 0.36 and 0.41 on the MET dataset (test set of n 1/4 2000 videos segments). These numbers are similar to inter-coder reliability of human coders. Finally, using Graph Convolutional Networks we make early strides (AUC=0.70) toward predicting the specific moments (45-90sec clips) when the PC is particularly weak/strong. Our findings inform the design of automatic classroom observation and also more general video activity recognition and summary recognition systems.
引用
收藏
页码:664 / 679
页数:16
相关论文
共 71 条
[21]  
Gordon G, 2016, AAAI CONF ARTIF INTE, P3951
[22]  
Grafsgaard J., 2013, PROC 6 INT C ED DATA
[23]  
Hamre B., 2018, PROC NAT RES C EARLY
[24]  
Hannun A, 2014, Arxiv, DOI arXiv:1412.5567
[25]   Student Learning Benefits of a Mixed-Reality Teacher Awareness Tool in AI-Enhanced Classrooms [J].
Holstein, Kenneth ;
McLaren, Bruce M. ;
Aleven, Vincent .
ARTIFICIAL INTELLIGENCE IN EDUCATION, PART I, 2018, 10947 :154-168
[26]   Automated Classification of Classroom Climate by Audio Analysis [J].
James, Anusha ;
Chua, Yi Han Victoria ;
Maszczyk, Tomasz ;
Nunez, Ana Moreno ;
Bull, Rebecca ;
Lee, Kerry ;
Dauwels, Justin .
9TH INTERNATIONAL WORKSHOP ON SPOKEN DIALOGUE SYSTEM TECHNOLOGY, 2019, 579 :41-49
[27]  
James A, 2018, PR IEEE INT CONF TEA, P983, DOI 10.1109/TALE.2018.8615327
[28]  
Jeni L.A., 2013, Proc. 10th IEEE Int. Conf. Workshops Autom. Face Gesture Recognit, P1
[29]  
Jiang HZ, 2017, IEEE INT CONF AUTOMA, P650, DOI [10.1109/MWSYM.2017.8058653, 10.1109/FG.2017.82]
[30]  
Kane T.J., 2013, Research Paper