Automatic granary sweeping strategy using visual large language model

Food security is a fundamental element of human survival. Reducing grain losses and ensuring grain quality have extremely important practical implications. Enhancing the granary's intelligence is particularly important due to several issues affecting residue grain sweeping, including manual inefficiency, incomplete coverage, and expensive equipment. This work proposes a new method called the Residual Grain Sweeping Visual Large Mode (RGSVLM)1 based on the Visual Large Language Model (VLLM). First, we constructed a semantic dataset containing images of various residual grain dispersal patterns captured in real granary environments. We also introduced an improved version of the Fast Segment Anything Model (FastSAM) algorithm to detect residual grains in the field images, extract visual features, and achieve accurate segmentation. In addition, we crafted prompt engineering that combines image data to produce corresponding textual datasets that effectively reflect the real-world situation. Next, we integrated this dataset with a chain of reasoning framework to fine-tune the visual large language model for specific tasks. This approach compensates for the original model's limitations in logical reasoning, enabling it to simulate human thought processes and generate clear and reasonable answers. In a granary environment, RGSVLM performs better than other models. This study's development and implementation of RGSVLM offers innovative concepts and techniques for building intelligent granaries.