Mobile Recognition and Tracking of Objects in the Environment through Augmented Reality and 3D Audio Cues for People with Visual Impairments

被引:6
|
作者
Kaul, Oliver Beren [1 ]
Behrens, Kersten [1 ]
Rohs, Michael [1 ]
机构
[1] Leibniz Univ Hannover, Hannover, Lower Saxony, Germany
来源
EXTENDED ABSTRACTS OF THE 2021 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS (CHI'21) | 2021年
关键词
Mobile Scene Recognition; Accessibility; Visually Impaired; Spatial Audio; Text-to-Speech; Auditory Scene Description; Object Detection; Collision Warnings;
D O I
10.1145/3411763.3451611
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
People with visual impairments face challenges in scene and object recognition, especially in unknown environments. We combined the mobile scene detection framework Apple ARKit with MobileNetv2 and 3D spatial audio to provide an auditory scene description to people with visual impairments. The combination of ARKit and MobileNet allows keeping recognized objects in the scene even if the user turns away from the object. An object can thus serve as an auditory landmark. With a search function, the system can even guide the user to a particular item. The system also provides spatial audio warnings for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can fnd items without visual feedback using the proposed application. The study also reveals that the range of local object detection through MobileNet-v2 was insufcient, which we aim to overcome using more accurate object detection frameworks in future work (YOLOv5x).
引用
收藏
页数:7
相关论文
共 50 条
  • [41] 3D modeling with the Tinmith mobile outdoor augmented reality system
    Piekarski, W
    IEEE COMPUTER GRAPHICS AND APPLICATIONS, 2006, 26 (01) : 14 - 17
  • [42] Detection, Classification and Tracking of Moving Objects in a 3D Environment
    Azim, Asma
    Aycard, Olivier
    2012 IEEE INTELLIGENT VEHICLES SYMPOSIUM (IV), 2012, : 802 - 807
  • [43] Tracking objects in an indoor environment using 3D models
    Nguyen, Congdu
    Le, Minh Tuan
    Kim, Hae-Kwang
    12TH INTERNATIONAL MULTI-MEDIA MODELLING CONFERENCE PROCEEDINGS, 2006, : 434 - 437
  • [44] A 3D Interaction Technique for Selection and Manipulation Distant Objects in Augmented Reality
    Bellarbi, Abdelkader
    Zenati, Nadia
    Otmane, Samir
    Belghit, Hayet
    Benbelkacem, Samir
    Messaci, Assia
    Hamidia, Mahfoud
    2017 5TH INTERNATIONAL CONFERENCE ON ELECTRICAL ENGINEERING - BOUMERDES (ICEE-B), 2017,
  • [45] Interactions with 3D virtual objects in augmented reality using natural gestures
    Dash, Ajaya Kumar
    Balaji, Koniki Venkata
    Dogra, Debi Prosad
    Kim, Byung-Gyu
    VISUAL COMPUTER, 2024, 40 (09): : 6449 - 6462
  • [46] Mobile Diminished Reality for Preserving 3D Visual Privacy
    Tabet, Salam
    Kayssi, Ayman
    Elhajj, Imad H.
    2023 INTERNATIONAL CONFERENCE ON INTELLIGENT METAVERSE TECHNOLOGIES & APPLICATIONS, IMETA, 2023, : 195 - 201
  • [47] 3DARModeler: A 3D modeling system in Augmented Reality environment
    Do, Trien V.
    Lee, Jong-Weon
    World Academy of Science, Engineering and Technology, 2009, 39 : 538 - 547
  • [48] An experimental study on the role of 3D sound in augmented reality environment
    Zhou, ZY
    Cheok, AD
    Yang, XB
    Qiu, Y
    INTERACTING WITH COMPUTERS, 2004, 16 (06) : 1043 - 1068
  • [49] 3D artificial objects recognition under virtual environment
    Yi, M
    Wang, PS
    15TH INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION, VOL 1, PROCEEDINGS: COMPUTER VISION AND IMAGE ANALYSIS, 2000, : 995 - 998
  • [50] Touch and hand gesture-based interactions for directly manipulating 3D virtual objects in mobile augmented reality
    Minseok Kim
    Jae Yeol Lee
    Multimedia Tools and Applications, 2016, 75 : 16529 - 16550