CottonSense: A high-throughput field phenotyping system for cotton fruit segmentation and enumeration on edge devices

High-throughput phenotyping (HTP) has become a powerful tool for gaining insights into the genetic and environmental factors that affect cotton (Gossypium spp.) growth and yield. With the recent advances in the field of computer vision, namely the integration of deep learning algorithms, the accuracy and efficiency of HTP systems have improved dramatically, enabling them to automatically quantify such fundamental phenotypic traits as fruit identification and enumeration. However, there is currently no HTP system available for counting all the reproductive phases of cotton crop that can be deployed in agronomic field conditions throughout the growing season. This study presents CottonSense, an advanced HTP system that overcomes the challenges of deployment across multiple growth periods by effectively segmenting and enumerating cotton fruits at four stages of growth, including square, flower, closed boll, and open boll. Consequently, CottonSense enhances agronomic management through increased opportunities for data collection and analysis. Using RGB-D cameras, it captures and processes both two and three-dimensional data, facilitating a wider range of phenotypic trait extractions such as crop biomass and plant architecture. To segment the cotton fruits, a Mask-RCNN model is trained and optimized for faster inference using TensorRT. The model yields an average AP score of 79% in segmentation across the four fruit categories. Moreover, the model's accuracy in estimating total fruit count per image is validated by a strong agreement with the counts given by ten domain experts, as reflected by an R2 value of 0.94. Furthermore, to accurately count the segmented fruits over large populations of plants, an enumeration algorithm based on a tracking strategy is developed that achieves an R2 value of 0.93 when compared to handcounted fruits in the field. The proposed HTP system, which is implemented entirely on an edge computing device, is cost-effective and power-efficient, making it an effective tool for high-yield cotton breeding and crop improvement. The code for CottonSense is publicly available at https://github.com/FeriBolour/CottonSense.