Environment awareness, multimodal interaction, and intelligent assistance in industrial augmented reality solutions with deep learning

Augmented reality is increasingly used in various fields, especially industrial applications. Although augmented reality devices’ characteristics and technological benefits are still evolving, augmented reality’s clear advantages in facilitating mechanical tasks and improving operator performance have made it popular. In industrial settings, the human factor remains irreplaceable, but the evolution of artificial intelligence has allowed any activity on the shop floor to be given new semantic possibilities. Through a semantic layer, it is possible to interpret and validate the environment, provide multimodal interaction, and analyze and evaluate information to detect anomalies or risky situations. Deep learning has opened up new possibilities for existing augmented reality solutions, such as visual interpretation of the environment, natural language understanding for problem-solving, or automatic anomaly detection. This new intelligent layer minimizes unnecessary interactions with the environment, validates the operator’s actions, and increases comfort, safety, and focus, making them more efficient in high cognitive level tasks. This work presents a general architecture based on a Semantic layer that relies on augmented reality systems and validates its advantages in a real industrial setting. Overall, integrating artificial intelligence and augmented reality solutions in industrial settings offers significant potential for improving productivity, safety, and worker satisfaction.