A Dexterous Hand-Arm Teleoperation System Based on Hand Pose Estimation and Active Vision

被引:11
作者
Li, Shuang [1 ]
Hendrich, Norman [1 ]
Liang, Hongzhuo [1 ]
Ruppel, Philipp [1 ]
Zhang, Changshui [2 ]
Zhang, Jianwei [1 ]
机构
[1] Univ Hamburg, Dept Informat, Tech Aspects Multimodal Syst Grp, D-20146 Hamburg, Germany
[2] Tsinghua Univ, Beijing Natl Res Ctr Informat Sci & Technol, Dept Automat, State Key Lab Intelligent Technol & Syst, Beijing 100190, Peoples R China
基金
美国国家科学基金会;
关键词
Robots; Pose estimation; Cameras; Robot sensing systems; Manipulators; Robot vision systems; Machine vision; Active vision; deep neural network; dexterous hand-arm system; hand pose estimation; vision-based teleoperation; RECOGNITION; ALGORITHMS;
D O I
10.1109/TCYB.2022.3207290
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Markerless vision-based teleoperation that leverages innovations in computer vision offers the advantages of allowing natural and noninvasive finger motions for multifingered robot hands. However, current pose estimation methods still face inaccuracy issues due to the self-occlusion of the fingers. Herein, we develop a novel vision-based hand-arm teleoperation system that captures the human hands from the best viewpoint and at a suitable distance. This teleoperation system consists of an end-to-end hand pose regression network and a controlled active vision system. The end-to-end pose regression network (Transteleop), combined with an auxiliary reconstruction loss function, captures the human hand through a low-cost depth camera and predicts joint commands of the robot based on the image-to-image translation method. To obtain the optimal observation of the human hand, an active vision system is implemented by a robot arm at the local site that ensures the high accuracy of the proposed neural network. Human arm motions are simultaneously mapped to the slave robot arm under relative control. Quantitative network evaluation and a variety of complex manipulation tasks, for example, tower building, pouring, and multitable cup stacking, demonstrate the practicality and stability of the proposed teleoperation system.
引用
收藏
页码:1417 / 1428
页数:12
相关论文
共 49 条
[1]  
Ajili I, 2017, IEEE ROMAN, P1115, DOI 10.1109/ROMAN.2017.8172443
[2]   AUTOMATED TRACKING AND GRASPING OF A MOVING OBJECT WITH A ROBOTIC HAND EYE SYSTEM [J].
ALLEN, PK ;
TIMCENKO, A ;
YOSHIMI, B ;
MICHELMAN, P .
IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION, 1993, 9 (02) :152-165
[3]  
Aloimonos J., 1987, International Journal of Computer Vision, V1, P333, DOI 10.1007/BF00133571
[4]  
[Anonymous], Shadow robot
[5]  
[Anonymous], BIOTAC MANUAL
[6]  
[Anonymous], ULTRALEAP
[7]  
[Anonymous], PR2 MANUAL
[8]   Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation [J].
Antotsiou, Dafni ;
Garcia-Hernando, Guillermo ;
Kim, Tae-Kyun .
COMPUTER VISION - ECCV 2018 WORKSHOPS, PT VI, 2019, 11134 :287-301
[9]   Viewpoint optimization for aiding grasp synthesis algorithms using reinforcement learning [J].
Calli, B. ;
Caarls, W. ;
Wisse, M. ;
Jonker, P. .
ADVANCED ROBOTICS, 2018, 32 (20) :1077-1089
[10]  
Calli B, 2011, IEEE INT C INT ROBOT, P995, DOI 10.1109/IROS.2011.6048739