MixSegNext: A CNN-Transformer hybrid model for semantic segmentation and picking point localization algorithm of Sichuan pepper in natural environments

Precise identification of Sichuan pepper picking points is a prerequisite for the robotic harvesting of the crop. Picking robots typically operate in open, dynamic natural environments, which demands robustness in the Sichuan pepper picking point localization algorithm. Generally, the growth environment of Sichuan pepper is complex, and the growth posture varies. The branches of the pepper clusters are similar to the pepper branches, which can easily lead to misjudgment and omission in the localization process, making accurate visual picking point localization challenging. To rapidly and accurately locate target Sichuan pepper picking points in natural environments, this paper proposes a Sichuan pepper segmentation model and picking point localization algorithm based on MixSegNext. The algorithm is divided into three main parts. First, the MixSegNext network performs semantic segmentation on Sichuan pepper clusters and fruits to extract the picking targets. Then, by subtracting the extracted pepper fruit mask from the pepper cluster mask, the Sichuan pepper branch mask is obtained, and the main pepper branch mask is acquired through morphological operations and maximal connectivity analysis. Finally, edge extraction is performed on the main pepper branch mask, and the picking point is determined by finding the intersection between the central line of the contour and the edge. This paper compares MixSegNext with typical semantic segmentation networks and conducts picking point localization experiments. The results show that the network has better segmentation precision and high picking point localization accuracy. Furthermore, this paper deploys the network on embedded devices to perform Sichuan pepper inference segmentation, verifying the application effect of the algorithm, which can provide a reference for the visual positioning system of Sichuan pepper-picking robots.