Control framework for collaborative robot using imitation learning-based teleoperation from human digital twin to robot digital twin*

Despite the deployment of collaborative robots for various industrial processes, their teaching and control remain comparatively difficult tasks compared with general industrial robots. Various imitation learning methods involving the transfer of human poses to a collaborative robot have been proposed. However, most of these methods depend heavily on deep learning-based human recognition algorithms that fail to recognize complicated human poses. To address this issue, we propose an automated/semi-automated vision-based teleoperation framework using human digital twin and a collaborative robot digital twin models. First, a human pose is recognized and reasoned to a human skeleton model using a convolution encoder-decoder architecture. Next, the developed human digital twin model is taught using the skeletons. As human and collaborative robots have different joints and rotation architectures, pose mapping is achieved using the proposed Bezier curve-based smooth approximation. Then, a real collaborative robot is controlled using the developed robot digital twin. Furthermore, the proposed framework works successfully using a human digital twin in the case of recognition failures of human poses. To verify the effectiveness of the proposed framework, transfers of several human poses to a real collaborative robot are tested and analyzed.