Mobile Recognition and Tracking of Objects in the Environment through Augmented Reality and 3D Audio Cues for People with Visual Impairments

被引:6
|
作者
Kaul, Oliver Beren [1 ]
Behrens, Kersten [1 ]
Rohs, Michael [1 ]
机构
[1] Leibniz Univ Hannover, Hannover, Lower Saxony, Germany
来源
EXTENDED ABSTRACTS OF THE 2021 CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS (CHI'21) | 2021年
关键词
Mobile Scene Recognition; Accessibility; Visually Impaired; Spatial Audio; Text-to-Speech; Auditory Scene Description; Object Detection; Collision Warnings;
D O I
10.1145/3411763.3451611
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
People with visual impairments face challenges in scene and object recognition, especially in unknown environments. We combined the mobile scene detection framework Apple ARKit with MobileNetv2 and 3D spatial audio to provide an auditory scene description to people with visual impairments. The combination of ARKit and MobileNet allows keeping recognized objects in the scene even if the user turns away from the object. An object can thus serve as an auditory landmark. With a search function, the system can even guide the user to a particular item. The system also provides spatial audio warnings for nearby objects and walls to avoid collisions. We evaluated the implemented app in a preliminary user study. The results show that users can fnd items without visual feedback using the proposed application. The study also reveals that the range of local object detection through MobileNet-v2 was insufcient, which we aim to overcome using more accurate object detection frameworks in future work (YOLOv5x).
引用
收藏
页数:7
相关论文
共 50 条
  • [31] 3D registration based perception in augmented reality environment
    Chen, Heen
    He, Hanwu
    Mo, Jianqing
    Li, Jinfang
    Yang, Xian
    COGENT ENGINEERING, 2016, 3 (01):
  • [32] A novel interface for device diagnostics using speech recognition, augmented reality visualization, and 3D audio auralization
    Behringer, R
    Chen, S
    Sundareswaran, V
    Wang, K
    Vassiliou, M
    IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA COMPUTING AND SYSTEMS, PROCEEDINGS VOL 1, 1999, : 427 - 432
  • [33] Novel interface for device diagnostics using speech recognition, augmented reality visualization, and 3D audio auralization
    Behringer, Reinhold
    Chen, Steven
    Sundareswaran, Venkataraman
    Wang, Kenneth
    Vassiliou, Marius
    International Conference on Multimedia Computing and Systems -Proceedings, 1999, 1 : 427 - 432
  • [34] 3D Tracking in Unknown Environments Using On-Line Keypoint Learning for Mobile Augmented Reality
    Schall, Gerhard
    Grabner, Helmut
    Grabner, Michael
    Wohlhart, Paul
    Schmalstieg, Dieter
    Bischof, Horst
    2008 IEEE COMPUTER SOCIETY CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, VOLS 1-3, 2008, : 1333 - 1340
  • [35] Improving Visual Perception of Augmented Reality on Mobile Devices with 3D Red-Cyan Glasses
    Gungor, Cengiz
    Kurt, Murat
    2014 22ND SIGNAL PROCESSING AND COMMUNICATIONS APPLICATIONS CONFERENCE (SIU), 2014, : 1706 - 1709
  • [36] Distributed device diagnostics system utilizing augmented reality and 3D audio
    Rockwell Science Center, 1049 Camino Dos Rios, Thousand Oaks, CA 91360, United States
    Comput Graphics (Pergamon), 6 (821-825):
  • [37] A distributed device diagnostics system utilizing augmented reality and 3D audio
    Behringer, R
    Chen, S
    Sundareswaran, V
    Wang, K
    Vassiliou, M
    COMPUTERS & GRAPHICS-UK, 1999, 23 (06): : 821 - 825
  • [38] See-through 3D technology for augmented reality
    Lee, Byoungho
    Lee, Seungjae
    Li, Gang
    Jang, Changwon
    Hong, Jong-Young
    DIGITAL OPTICAL TECHNOLOGIES 2017, 2017, 10335
  • [39] Slimmer: Accelerating 3D Semantic Segmentation for Mobile Augmented Reality
    Zhang, Huanle
    Han, Bo
    Ip, Cheuk Yiu
    Mohapatra, Prasant
    2020 IEEE 17TH INTERNATIONAL CONFERENCE ON MOBILE AD HOC AND SMART SYSTEMS (MASS 2020), 2020, : 603 - 612
  • [40] Mobile 3D augmented-reality system for ultrasound applications
    Palmer, Cameron Lowell
    Haugen, Bjorn Olav
    Tegnander, Eva
    Eik-Nes, Sturla H.
    Torp, Hans
    Kiss, Gabriel
    2015 IEEE INTERNATIONAL ULTRASONICS SYMPOSIUM (IUS), 2015,