Large language model-based approach for human-mobile inspection robot interactive navigation

被引：0

作者：

Wang T. ^{[1
]}

Fan J. ^{[1
]}

Zheng P. ^{[1
]}

机构：

[1] Department of Industrial and Systems Engineering, The Hong Kong Polytechnic University

来源：

Jisuanji Jicheng Zhizao Xitong/Computer Integrated Manufacturing Systems, CIMS | 2024年 / 30卷 / 05期

关键词：

human-robot interaction; Industry; 5.0; large language model; smart manufacturing; vision and language navigation;

D O I：

10.13196/j.cims.2024.0139

中图分类号：

学科分类号：

摘要：

In the manufacturing field,the wide application of mobile robots has become the key to improving operational safety and efficiency.However,most existing robotic systems can only complete predefined navigation tasks,and cannot be adapted to the unstructured environment.To overcome this bottleneck,an interactive navigation method for mobile inspection robots based on large language models was introduced,which replaced operators in conducting inspections within hazardous industrial areas,and to execute complex navigation tasks based on verbal instructions.The High-Resolution Net (HRNet) model was utilized for semantic scene segmentation,integrating the segmentation results into the reconstructed 3D scene mesh during the point cloud fusion phase to create a comprehensive 3D semantic map.A large language model was used to make the robot comprehend human natural language instructions and generate Python code based on the 3D semantic map to complete navigation tasks.A series of experiments had been conducted to validate the effectiveness of the proposed system. © 2024 CIMS. All rights reserved.

引用

页码：1587 / 1594

页数：7

共 19 条

[1]

ZHAO Shanshan, Ml Zengzhen, CHEN Ren, Moving target detection and 3D reconstruction based on binocular vision[J/OL], Computer Integrated Manufacturing Systems, (2024)

[2]

MATHEW R, HIREMATH S S., Control of velocity-constrained stepper motor-driven Hilare robot for waypoint navigation, Engineering, 4, 4, pp. 491-499, (2018)

[3]

YUAN C, XIONG B, LI X Q, Et al., A novel intelligent inspection robot with deep stereo vision for three-dimensional concrete damage detection and quantification, Structural Health Monitoring, 21, 3, pp. 788-802, (2022)

[4]

WANG T, ZHENG P, LI S F, Et al., Multimodal human-robot interaction for human-centric smart manufacturing: A sur-vey, Advanced Intelligent Systems, 6, 3, (2024)

[5]

NIU Kai, WANG Peng, Survey on the research progress and development trend of vision-and-language navigation[J], Journal of Computer-Aided Design & Computer Graphics, 34, 12, (2022)

[6]

JEONG S, KIM M G, PARK J Y, Et al., Long-term monitoring method for tunnel structure transformation using a 3D light detection and ranging equipped in a mobile robot, Structural Health Monitoring, 22, 6, pp. 3742-3760, (2023)

[7]

CHEAH W, GROVES K, MARTIN H, Et al., MIRRAX: A reconfigurable robot for limited access environments, IEEE Transactions on Robotics, 39, 2, pp. 1341-1352, (2022)

[8]

KAKOGAWA A, MAS ., An in-pipe inspection module with an omnidirectional bent-pipe self-adaptation mechanism using a joint torque control, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4347-4352, (2019)

[9]

LENG J W, SHA W N, WANG B C, Et al., Industry 5. 0: Prospect and retrospect, Journal of Manufacturing Systems, 65, pp. 279-295, (2022)

[10]

MIAO Xiu, HOU Wenjun, Human-computer interaction multi-task modeling based on implicit intent EEG decoding[J/OL], Computer Integrated Manufacturing Systems

← 1 2 →