Improved binocular localization of kiwifruit in orchard based on fruit and calyx detection using YOLOv5x for robotic picking

Localization is the first critical step for picking robots to successfully grasp fruit. However, classical binocular localization methods adopted global matching for kiwifruit, which may result in a large amount of mismatching feature points in complex orchard and thus cause low localization accuracy. Therefore, an improved binocular localization method of calyxes based on deep learning was proposed to accurately detect and locate kiwifruit for robotic harvesting. Calyxes in the binocular images and kiwifruit in the left images of the binocular images were detected using You Only Look Once version 5 xlarge (YOLOv5x). The detected calyxes were matched in the binocular images using kiwifruit and calyx pairing and kiwifruit matching. The matched calyxes in the binocular images were used to locate calyxes using three localization methods. Specifically, three binocular localization methods, i.e., calyx localization (CL), fruit localization (FL), and depth information from depth map (DIDM), were compared to find the optimal one. Ground truth three-dimensional coordinates of calyxes was measured by laser rangefinder and coordinate paper on a self-designed experimental platform. Results showed that YOLOv5x achieved an average precision (AP) of 99.5 % on kiwifruit detection and a mean AP of 93.5 % on kiwifruit and calyx detection with a detection speed of 108 ms per image. Average deviation of X-axis, Y-axis, and Z-axis obtained by the CL method were 7.9 mm, 6.4 mm, and 4.8 mm, respectively. Compared with the FL and DIDM methods, localization error rate of the proposed CL method was reduced by 55.1 % and 53.8 %, respectively. It indicates that the proposed CL method is promising for robotic harvesting.