Artificial intelligence (AI) has emerged as a powerful tool, enabling the effective study of reef fish populations and their dynamic interactions within marine ecosystems. In coral reef conservation, accurate detection and tracking of fish species are essential for assessing biodiversity and ecosystem health. These data-driven insights support Marine Spatial Planning (MSP), a strategic framework for the spatial organization and sustainable management of ocean resources. By integrating AI-driven analyses, MSP can facilitate evidence-based decision-making, enhance conservation efforts, and promote the sustainable use of marine environments. Automated fish detection and tracking, combined with visual species identification and georeferencing, provide essential data for Marine Spatial Planning (MSP) mapping systems. This approach facilitates accurate species counting for population evaluation, enables fish behavior analysis, supports ecosystem monitoring, and enhances the understanding of species interactions within their environment. To address this, the paper proposes an improved You Only Look Once (YOLO) v5 with Deep Simple Online Real-time Tracking (SORT) technique to perform species detection, tracking, and behavioral pattern analysis of the non-cryptic fish species. The proposed methodology comprises three modules: Pre-trained COCO Dataset model, YOLOv5 and DeepSORT module to detect and track non-cryptic reef fish species. To improve the accuracy of the detection model, the transfer learning technique is used. The experiments are carried out to evaluate the performance of these models on underwater video sequences. The quantitative evaluation is done using metrics such as precision, recall, accuracy, Multiple Object Tracking Accuracy (MOTA) and Multiple Objects Tracking Precision (MOTP) to assess the detection and tracking algorithm. It is observed that the proposed methodology—improved YOLOv5 with DeepSORT after transfer learning technique has shown superior performance and achieved detection accuracy of 96%.