Deep learning-based augmented reality work instruction assistance system for complex manual assembly

被引:6
作者
Li, Wang [1 ,2 ]
Xu, Aibo [2 ]
Wei, Ming [2 ]
Zuo, Wei [2 ]
Li, Runsheng [2 ]
机构
[1] Huazhong Univ Sci & Technol, Sch Mech Sci & Engn, Wuhan 430074, Peoples R China
[2] Wuhan Fiberhome Tech Serv Co Ltd, Wuhan 430205, Peoples R China
关键词
Assembly; Augmented reality; Deep learning; Mark -less registration; Quality inspection; INSPECTION; COLLABORATION; REGISTRATION; ALGORITHM;
D O I
10.1016/j.jmsy.2024.02.009
中图分类号
T [工业技术];
学科分类号
08 ;
摘要
The manual assembly process of complex products is lengthy, the assembly requirements are difficult to recall, and the assembly quality requirements are high. The separation of the operation guidance information from the real physical object in the traditional paper manual easily distracts the operator, increasing their cognitive burden. To address this issue, we integrate augmented reality (AR) and artificial intelligence (AI) technologies for manual assembly assistance. A novel encoding and decoding convolutional neural network (CNN) is built to realize accurate AR registration under mark-less assembly environment. An assembly quality inspection method integrating a neural network and virtual model matching was proposed for AR systems. The specific process of the proposed method includes two stages: offline and online. In the first stage, a monocular RGB image dataset was built for assembling object keypoints and assembly object detection. CNN models were designed and trained for deployment in an AR-assisted assembly system. Second, the deployed CNN models were used to perform AR registration and assembly quality inspection. Experimental results demonstrate that the proposed method can accurately present AR work instruction guidance and assembly quality inspection for manual assembly. The proposed AR registration method can achieve a frame rate of 25FPS, which satisfies the timeliness requirements of the AR system. The accuracy rate of the proposed assembly quality inspection method for the MONA component assembly reached 92.5%.
引用
收藏
页码:307 / 319
页数:13
相关论文
共 62 条
[1]   Deep learning methods for object detection in smart manufacturing: A survey [J].
Ahmad, Hafiz Mughees ;
Rahimi, Afshin .
JOURNAL OF MANUFACTURING SYSTEMS, 2022, 64 :181-196
[2]   SURF: Speeded up robust features [J].
Bay, Herbert ;
Tuytelaars, Tinne ;
Van Gool, Luc .
COMPUTER VISION - ECCV 2006 , PT 1, PROCEEDINGS, 2006, 3951 :404-417
[3]   Automatic Inspection of Aeronautical Mechanical Assemblies by Matching the 3D CAD Model and Real 2D Images [J].
Ben Abdallah, Hamdi ;
Jovancevic, Igor ;
Orteu, Jean-Jose ;
Brethes, Ludovic .
JOURNAL OF IMAGING, 2019, 5 (10)
[4]   Solder Paste Scooping Detection by Multilevel Visual Inspection of Printed Circuit Boards [J].
Benedek, Csaba ;
Krammer, Oliver ;
Janoczki, Mihaly ;
Jakab, Laszlo .
IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, 2013, 60 (06) :2318-2331
[5]  
Berndt D., Digital assembly inspection: automatic quality control even for small quantities
[6]  
Blattner J, 2021, SSRN Electron J, DOI [10.2139/ssrn.3858632, DOI 10.2139/SSRN.3858632]
[7]   Ageing workforce management in manufacturing systems: state of the art and future research agenda [J].
Calzavara, Martina ;
Battini, Daria ;
Bogataj, David ;
Sgarbossa, Fabio ;
Zennaro, Ilenia .
INTERNATIONAL JOURNAL OF PRODUCTION RESEARCH, 2020, 58 (03) :729-747
[8]   Mechanical Assembly Monitoring Method Based on Depth Image Multiview Change Detection [J].
Chen, Chengjun ;
Li, Changzhi ;
Li, Dongnian ;
Zhao, Zhengxu ;
Hong, Jun .
IEEE TRANSACTIONS ON INSTRUMENTATION AND MEASUREMENT, 2021, 70
[9]   Wavelet transform based image template matching for automatic component inspection [J].
Cho, Han-Jin ;
Park, Tae-Hyoung .
INTERNATIONAL JOURNAL OF ADVANCED MANUFACTURING TECHNOLOGY, 2010, 50 (9-12) :1033-1039
[10]   A Real-Time Physical Progress Measurement Method for Schedule Performance Control Using Vision, an AR Marker and Machine Learning in a Ship Block Assembly Process [J].
Choi, Taihun ;
Seo, Yoonho .
SENSORS, 2020, 20 (18) :1-25