Coupling AR with Object Detection Neural Networks for End-User Engagement

The mobile Augmented Reality (AR) technology offers an accessible, inexpensive, and rich user experience that has the potential to engage end-users to an immersive environment. Machine learning (ML) algorithms can be tailored and tuned to carry out a large variety of tasks pertaining to data coming from highly specific environments, thereby improving the decision-making process, and uncovering gaps and opportunities in a wide range of applied fields. High data availability and modern algorithms contribute to the changing way end-users interact with their environment to obtain information, train, and socialize. In this study, we leverage the widespread adoption of mobile phones in daily lives and their advanced features (such as connectivity and location-awareness along with powerful optical and image sensors) to design, develop and test, anAR application that offers a unique user-environment interaction through object detection functionalities. The intended use of the mobile AR application is to foster end-user engagement, enrich user experiences, improve self-efficacy and motivation, and contribute to dissemination and communication activities. Due to its technical and gamification features, and the ability to customize its content through a web-based Content Management Service, it has great potential to be exploited in a variety of contexts including education and training, touristic and cultural appreciation, art and heritage promotion, among others. This paper presents the development and testing of the mobile application and details its architecture and the pertaining implementation challenges. The design features and implementation details of our approach result in a 90% detection accuracy for common objects. Such a performance contributes to the discourse on the use of mobile AR coupled with ML functionalities as a tool stimulating end-user engagement and hands-on learning by facilitating environment exploration, and helping end-users to move from passive observation to creative interaction.