Multimodal interaction design and application in augmented reality for chemical experiment

被引：1
作者：
Xiao M. [1 ,2 ]
Feng Z. [1 ,2 ]
Yang X. [1 ,2 ]
Xu T. [1 ,2 ]
Guo Q. [1 ,2 ]
机构：
[1] Department of Information Science and Engineering, University of Jinan, Jinan
[2] Shandong Provincial Key Laboratory of Network Based Intelligent Computing, Jinan
来源：
Virtual Reality and Intelligent Hardware | 2020年 / 2卷 / 04期
关键词：
Augmented reality; Augmented Reality Chemistry Lab; Gesture recognition; Intelligent equipment; Multimodal Interaction;
D O I：
10.1016/j.vrih.2020.07.005
中图分类号：
学科分类号：
摘要：
Background: Augmented reality classrooms have become an interesting research topic in the field of education, but there are some limitations. Firstly, most researchers use cards to operate experiments, and a large number of cards cause difficulty and inconvenience for users. Secondly, most users conduct experiments only in the visual modal, and such single-modal interaction greatly reduces the users' real sense of interaction. In order to solve these problems, we propose the Multimodal Interaction Algorithm based on Augmented Reality (ARGEV), which is based on visual and tactile feedback in Augmented Reality. In addition, we design a Virtual and Real Fusion Interactive Tool Suite (VRFITS) with gesture recognition and intelligent equipment. Methods: The ARGVE method fuses gesture, intelligent equipment, and virtual models. We use a gesture recognition model trained by a convolutional neural network to recognize the gestures in AR, and to trigger a vibration feedback after a recognizing a fivefinger grasp gesture. We establish a coordinate mapping relationship between real hands and the virtual model to achieve the fusion of gestures and the virtual model. Results: The average accuracy rate of gesture recognition was 99.04%. We verify and apply VRFITS in the Augmented Reality Chemistry Lab (ARCL), and the overall operation load of ARCL is thus reduced by 29.42%, in comparison to traditional simulation virtual experiments. Conclusions: We achieve real-time fusion of the gesture, virtual model, and intelligent equipment in ARCL. Compared with the NOBOOK virtual simulation experiment, ARCL improves the users' real sense of operation and interaction efficiency. © 2019 Beijing Zhongke Journal Publishing Co. Ltd
引用
收藏
页码：291 / 304
页数：13
相关论文
共 22 条
  • [1] Collazos C.A., Merchan L., Human-computer interaction in Colombia: bridging the gap between education and industry, IT Professional, 17, 1, pp. 5-9, (2015)
  • [2] Desai K., Belmonte U.H.H., Jin R., Prabhakaran B., Diehl P., Ramirez V.A., Johnson V., Gans M., Experiences with multimodal collaborative virtual laboratory (MMCVL), 2017 IEEE Third International Conference on Multimedia Big. Data (BigMM), pp. 376-383, (2017)
  • [3] Chen L., Tang W., John N.W., Wan T.R., Zhang J.J., SLAM-based dense surface reconstruction in monocular Minimally. Invasive Surgery and its application to Augmented Reality, Computer Methods and Programs in Biomedicine, 158, pp. 135-146, (2018)
  • [4] Huynh B., Orlosky J., Hollerer T., In-situ labeling for augmented reality language learning, 2019 IEEE Conference on. Virtual Reality and 3D User Interfaces (VR), pp. 1606-1611, (2019)
  • [5] Karambakhsh A., Kamel A., Sheng B., Li P., Yang P., Feng D.D., Deep gesture interaction for augmented anatomy learning, International Journal of Information Management, 45, pp. 328-336, (2019)
  • [6] Sun C.X., The design and implementation of children's education system based on augmented reality, (2017)
  • [7] Fidan M., Tuncel M., Integrating augmented reality into problem based learning: the effects on learning achievement and. attitude in physics education, Computers & Education, 142, (2019)
  • [8] Dave I.R., Chaudhary V., Upla K.P., Simulation of analytical chemistry experiments on augmented reality platform, Advances in Intelligent Systems and Computing, pp. 393-403, (2018)
  • [9] Ibili E., Cat M., Resnyansky D., Sahin S., Billinghurst M., An assessment of geometry teaching supported with augmented. reality teaching materials to enhance students' 3D geometry thinking skills, International Journal of Mathematical. Education in Science and Technology, 51, 2, pp. 224-246, (2020)
  • [10] Rani S.S., Dhrisya K.J., Ahalyadas M., Hand gesture control of virtual object in augmented reality, 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 1500-1505, (2017)
123