A bucket fill factor estimation method in construction environments by fusing deep learning and machine vision

The need for high productivity in the construction field is becoming increasingly urgent, and the efficiency of loading and unloading materials by construction vehicles is one of the main influencing factors. The fill factor, which refers to the percentage of the volume of materials in the shovel bucket to the capacity of the shovel bucket, can significantly affect the productivity of construction vehicles, and thereby affect the environmental pollution caused by these vehicles. Therefore, an effective estimation of the fill factor is a prerequisite for improving productivity and suppressing pollution in construction vehicles. However, previous studies have been conducted under ideal environments, the actual harsh environments of the construction sites are not in the purview. Therefore, in this paper, the fill factor of a loader is estimated in nine construction environments, including six ideal environments and three harsh environments. First, the corresponding image processing pipelines are proposed for a variety of environmental characteristics. Then, a stereo segmentation map is constructed based on the loader bucket trajectory and the knowledge of different color space characteristics, which not only addresses the environmental limitations but also significantly improves the accuracy. Finally, an improved CNN is combined with probabilistic knowledge for the fill factor estimation, and transfer learning is also applied to improve the training speed and accuracy. The results show that a satisfactory accuracy of 96.89% is achieved for the abovementioned construction environments, which results in a more efficient and cleaner production of construction vehicles.