Towards end-to-end deep RNN based networks to precisely regress of the lettuce plant height by single perspective sparse 3D point cloud

Nowadays, 3D point cloud is supposed to be the most direct and effective data form for studying plant morphology structure. However, automatic and high-throughput acquisition of accurate individual plant height traits from 3D point cloud remains an urgent challenging problem. Summarizing the related research results in recent years, the factors limiting its application mainly come from these aspects: (1) Many existing methods require spatial auxiliary information such as ground control points (GCP), digital terrain models (DTM) and digital surface models (DSM) to obtain accurate plant height; (2) For 3D point cloud data in different environments, specialized modeling and careful parameter fine-tuning are usually required; (3) Sometimes, the point cloud processing involves the combined utilization of multiple programming languages and software, which is difficult for system integration. Focusing on these challenges, firstly, we proposed a novel end-to-end deep Recurrent Neural Network (RNN) based regression network framework called DRN, which consists of three parts: point cloud feature extraction network, deep RNN and regression network. The convolution operations-based point cloud feature extraction network is function as filtering noise, outliers and redundant information; The deep RNN network with long and short-term memory (LSTM) ability is used to learn the relationships between the feature points on the high-dimensional feature sequence separated by a certain distance; regression network is used to regress the output from deep RNN to plant height value. Experiments results on the 3rd Greenhouse Growing Challenge datasets show that DRN can directly regress the plant height of a single plant effectively without manual operations and the participation of spatial auxiliary information with an R2 of 0.948 and a relative root mean square error (RRMSE) of 10.06% in four different varieties of lettuce at different growth period. After studying the influence of the weights of the x, y, z coordinate of the input 3D point cloud on the regression result, then, we design a Dimension Attention (DA) module at the front end of the feature extraction network to learning the characteristic coordinate weight for every input point cloud sample. The DRN network with a DA module is called D-DRN, experiment results indicate D-DRN tend to achieve better result (R2 = 0.960; RRMSE = 8.680%) than DRN. Considering the end-to-end-based DRN and D-DRN network capable of ease of integration and their considerable prediction accuracy on public datasets, we believe they has a certain complementary effect on the existing study methods of obtaining plant morphological structure phenotype by point cloud data.