FishTrack: Multi-object tracking method for fish using spatiotemporal information fusion

Tracking the fish is a key step in analyzing fish behavior, evaluating their health levels, and warning of abnormal water quality, so it is of significant importance for intelligent monitoring in fish farming. However, multi-object tracking for fish is a very challenging task due to foam occlusion in the factory tank, high individual similarity, and fast-paced motion of fish. To address the above issues, an online multi-fish tracking model called FishTrack is proposed with 3 branches of target detection, track prediction, and re-identification, which establishes the motion model and appearance model for fish simultaneously, thus achieving the online multi-fish tracking. It is a complete Encoder-Decoder structure and the Pyramid Vision Transformer (PVT) is adopted as the backbone network to extract multi-level features. Then an Encoder is specially designed to encode the historical infor-mation of the positions of fish, and automatically update its spatiotemporal information in an autoregressive fashion, to perform the fusion of the spatiotemporal information of fish targets and avoid manual selection of spatiotemporal features. Finally, a parallel dual-Decoder is used to decode the motion and appearance features of fish to reduce the interference of the two optimization directions during the joint model training. The motion cues are first used in linear assignment to reduce the interference of occlusion and deformation issues, and when the motion model fails to track, the appearance cues can recover the identity to deal with long occlusion. The experiments on the established multiple fish tracking dataset showed that the higher order tracking accuracy (HOTA) reached 71.4%, the multi-object tracking accuracy (MOTA) reached 94.8%, the most tracked ratio (MT) reached 93.3%, and the identification F1 score (IDF1) reached 82.5%. The results show that FishTrack can solve the problem of tracking accuracy decrease caused by foam occlusion and fish deformation, and save the inference time by sharing features, increasing the robustness of online multi-fish tracking in factory farming.