Timely and accurate macadamia identification is the key to automated orchard harvesting. This study proposes an improved recognition method for the YOLOv7 model to address fruit overlap, leaf-trunk occlusion and small target problems in complex backgrounds. First, the novel BiFormer attention mechanism is introduced. This mechanism utilizes two-layer routing to achieve dynamic sparse attention, effectively reducing the number of computations, enhancing the perception of small-scale targets. Second, the asymptotic feature pyramid network (AFPN) replaces the YOLOv7 backbone network, reducing the semantic differences and parameters between different layers and improving small target detection in complex scenes. In addition, the repulsion loss function replaces the YOLOv7 default loss function to address dense fruit arrangement and fruit occlusion in the data set. This successfully reduces false detections. The validity of the model is verified by ablation and comparison experiments, which show that the improved YOLOv7 model achieves an average accuracy, precision, recall and F1 value of 99.7%, 99.6%, 99.3% and 99.4%, respectively. The average accuracy of the improved model increased by 7.5 percentage points compared with that of the YOLOv7 model. Overall, the improved YOLOv7 model can accurately recognize macadamia under complex lighting and background conditions with high detection accuracy and robustness.