A novel mixed reality remote collaboration system with adaptive generation of instructions

Mixed Reality (MR) remote collaboration supports a shared workspace and various non-verbal communication cues that enable remote experts to express ideas in an immersive Virtual Reality (VR) space to guide local workers in industrial physical tasks by creating Augmented Reality (AR) instructions, such as remote assembly/ disassembly guidance, emergency maintenance, and training. However, due to the information non-symmetrical caused by geographical distribution, it can be challenging and high-workload for remote experts to create clear and detailed AR instructions for local workers to match spatial relationships with the local workspace, while comprehensively considering the situation information (e.g., task-related objects and partner's environment) in complex industrial physical tasks. Advances in ambient intelligence bring ideas for building a new remote collaboration framework that supports the adaptive generation of instructions. In the study, we developed a novel MR remote collaboration prototype system, which can automatically and adaptively generate clear, detailed, and standardized AR instructions based on remote experts' simple and intuitive interactions and local contextual information to assist remote experts in guiding local workers to complete industrial physical tasks, especially industrial assembly tasks. The effect of two interfaces on industrial assembly tasks involving relatively complex process information and environments was explored in a user study: one was a baseline solution that uses a typical MR remote collaboration interface to support remote experts to create instructions (RECI) using popular non-verbal communication cues in the shared local 3D stereoscopic scene, and the other was our novel interface that additionally supported the adaptive generation of instructions via context awareness (AGICA) on top of the typical interface. The result of the user study showed that AGICA significantly improves collaboration performance, reduces errors and workload, and enhances usability and user experience compared to RECI. This demonstrates that supporting the adaptive generation of instructions is a feasible way to enhance MR remote collaboration in industrial assembly tasks with relatively complex process information and environments. Our research findings have a certain guiding significance for the design of MR remote collaboration systems in such tasks.