Augmented reality interaction: a comprehensive review of gesture and speech integration techniques

被引：0
作者：
Bai, Jin [1 ,2 ]
Sunar, Mohd Shahrizal [1 ,2 ]
Mohd Suaib, Norhaida [1 ]
机构：
[1] Faculty of Computing, Universiti Teknologi Malaysia, Skudai Johor
[2] Media and Game Innovation Centre of Excellence, Institute of Human Centered Engineering, Universiti Teknologi Malaysia, Skudai Johor
来源：
Neural Computing and Applications | 2025年 / 37卷 / 17期
关键词：
Augmented reality; Gesture; Interaction technique; Multimodal; Speech;
D O I：
10.1007/s00521-025-11190-w
中图分类号：
G4 [教育];
学科分类号：
04 ; 0401 ;
摘要：
In the ever-evolving landscape of Augmented Reality (AR), gesture and speech interaction technologies have emerged as pivotal components, reshaping experiences across diverse domains, from art to healthcare and education. Existing reviews may talk extensively about various types of interactions in augmented reality, but this paper fills a gap in this targeted area by discussing research that adopted both gesture and speech interactions. This paper employs the PRISMA methodology to curate and analyze a selection of cutting-edge research articles, offering a systematic and comprehensive review of 16 AR-based gesture, speech, and multimodal interaction technologies published between 2019 and 2023. Among them, “gesture + speech” accounted for 75%, while “gesture + speech + gaze” and “gesture + speech + head movement” models accounted for 12.5%. Highlighting the primary findings and contributions of this review, this article uncovers the prevailing trends in interaction technology implementation within AR environments. The review explores not only the methodologies but also the practical applications across a spectrum of AR scenarios. This comprehensive overview serves to contextualize the significance of these interaction technologies in enhancing user experiences and opens up new avenues for future research and development. Furthermore, this article underscores the real-world implications of these findings, shedding light on the potential for broader integration of gesture and speech in AR applications. As we look ahead, this paper provides insights into potential areas for further exploration in this dynamic field. By delving into the past, illuminating the present, and paving the way for the future, this review underscores the transformative power of gesture and speech interaction technologies in the realm of Augmented Reality. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2025.
引用
收藏
页码：11347 / 11377
页数：30
相关论文
共 79 条
[1]  
Aliprantis J., Konstantakis M., Nikopoulou R., Et al., Natural interaction in augmented reality context, VIPERC@ IRCDL, (2019)
[2]  
Hertel J., Karaosmanoglu S., Schmidt S., Et al., A taxonomy of interaction techniques for immersive augmented reality based on an iterative literature review, 2021 IEEE international symposium on mixed and augmented reality (ISMAR), (2021)
[3]  
Song Y., Koeck R., Luo S., Review and analysis of augmented reality (ar) literature for digital fabrication in architecture, Autom Constr, 128, (2021)
[4]  
LaViola J.J., Kruijff E., McMahan R.P., Et al., 3D user interfaces: theory and practice, (2017)
[5]  
Saroha K., Sharma S., Bhatia G., Human computer interaction: an intellectual approach, IJCSMS Int J Comput Sci Manag Stud, 11, 2, pp. 147-154, (2011)
[6]  
Irshad S., Rambli D.R.B.A., User experience of mobile augmented reality: A review of studies, 2014 3Rd International Conference on User Science and Engineering, (2014)
[7]  
Jackson P., Understanding understanding and ambiguity in natural language, Proc Comput Sci, 169, pp. 209-225, (2020)
[8]  
Kim M., Lee J.Y., Touch and hand gesture-based interactions for directly manipulating 3d virtual objects in mobile augmented reality, Multimed Tools Appl, 75, pp. 16529-16550, (2016)
[9]  
Nazri N.I.A.M., Rambli D.R.A., The roles of input and output modalities on user interaction in mobile augmented reality application, Proceedings of the Asia Pacific HCI and UX Design Symposium, (2015)
[10]  
Ling J., Peng Z., Yin L., Et al., How efficiency and naturalness change in multimodal interaction in mobile navigation apps, Advances in Usability, User Experience, Wearable and Assistive Technology: Proceedings of the AHFE 2020 Virtual Conferences on Usability and User Experience, Human Factors and Assistive Technology, Human Factors and Wearable Technologies, and Virtual Environments and Game Design, (2020)
12345678