Visual perception system design for rock breaking robot based on multi-sensor fusion

被引：6

作者：

Li, Jinguang ^{[1
]}

Liu, Yu ^{[1
]}

Wang, Shuai ^{[1
]}

Wang, Linwei ^{[1
]}

Sun, Yumeng ^{[1
]}

Li, Xin ^{[1
]}

机构：

[1] Northeastern Univ, Sch Mech Engn & Automat, Shenyang 110819, Peoples R China

来源：

MULTIMEDIA TOOLS AND APPLICATIONS | 2023年 / 83卷 / 8期

基金：

中国国家自然科学基金;

关键词：

Rock breaking; Visual perception; Multi-sensor fusion; 3D point cloud; MODEL; SHAPE; LIDAR;

D O I：

10.1007/s11042-023-16189-w

中图分类号：

TP [自动化技术、计算机技术];

学科分类号：

0812 ;

摘要：

In recent years, mining automation has received significant attention as a critical focus area. Rock breaking robots are commonly used equipment in the mining industry, and their automation requires an accurate and fast visual perception system. Currently, rock detection and determination of rock breaking surfaces heavily rely on operator experience. To address this, this paper leverages multi-sensor fusion techniques, specifically camera and lidar fusion, as the perception system for the rock breaking robot. The advanced PP-YOLO series algorithm is employed for 2D detection, enabling the generation of specific detection results based on the breaking requirements. Furthermore, 3D reconstruction of rocks detected in the 2D area is performed using point cloud data. The extraction of rock breaking surfaces is achieved through point cloud segmentation and statistical filtering methods. Experimental results demonstrate a rock detection speed of 13.8 ms and the mAP value of 91.2%. The segmentation accuracy for rock breaking surfaces is 75.46%, with an average recall of 91.08%. The segmentation process takes 73.09 ms, thus meeting the real-time detection and segmentation needs within the specified rock breaking range. This study effectively addresses the limitations associated with single sensor information.

引用

页码：24795 / 24814

页数：20

共 36 条

[21] Robust Rock Detection and Clustering with Surface Analysis for Robotic Rock Breaking Systems [J].

Lampinen, Santeri ;

Mattila, Jouni .

2021 IEEE/ASME INTERNATIONAL CONFERENCE ON ADVANCED INTELLIGENT MECHATRONICS (AIM), 2021, :140-147

[22] Autonomous robotic rock breaking using a real-time 3D visual perception system [J].

Lampinen, Santeri ;

Niu, Longchuan ;

Hulttinen, Lionel ;

Niemi, Jouni ;

Mattila, Jouni .

JOURNAL OF FIELD ROBOTICS, 2021, 38 (07) :980-1006

[23] A vision-based marker-less pose estimation system for articulated construction robots [J].

Liang, Ci-Jyun ;

Lundeen, Kurt M. ;

McGee, Wes ;

Menassa, Carol C. ;

Lee, SangHyun ;

Kamat, Vineet R. .

AUTOMATION IN CONSTRUCTION, 2019, 104 :80-94

[24] Research on Intelligent Identification of Rock Types Based on Faster R-CNN Method [J].

Liu, Xiaobo ;

Wang, Huaiyuan ;

Jing, Hongdi ;

Shao, Anlin ;

Wang, Liancheng .

IEEE ACCESS, 2020, 8 :21804-21812

[25] Detecting rocks in challenging mining environments using convolutional neural networks and ellipses as an alternative to bounding boxes [J].

Loncomilla, Patricio ;

Samtani, Pavan ;

Ruiz-del-Solar, Javier .

EXPERT SYSTEMS WITH APPLICATIONS, 2022, 194

[26]

Long X, 2020, Arxiv, DOI [arXiv:2007.12099, 10.48550/arXiv.2007.12099, DOI 10.48550/ARXIV.2007.12099]

[27] A method to predict shape and trajectory of charge in industrial mills [J].

Maleki-Moghaddam, M. ;

Yahyaei, M. ;

Banisi, S. .

MINERALS ENGINEERING, 2013, 46-47 :157-166

[28] Automatic Identification of Large Fragments in a Pile of Broken Rock Using a Time-of-Flight Camera [J].

McKinnon, Christopher ;

Marshall, Joshua A. .

IEEE TRANSACTIONS ON AUTOMATION SCIENCE AND ENGINEERING, 2014, 11 (03) :935-942

[29]

Misra D, 2020, Arxiv, DOI arXiv:1908.08681

[30]

Niu LC, 2019, IEEE INT CON AUTO SC, P1124, DOI [10.1109/coase.2019.8842859, 10.1109/COASE.2019.8842859]

← 1 2 3 4 →