The technological development in today’s era has unlocked the measures to propose new applications for the fruit industry. Automation boosts the economic growth and productivity of the country. Fruit quality detection in complex backgrounds using an automated system is significant for this sector. Fruit sorting has an impact on the export market and quality evaluation. One of the crucial qualities of grading fruits is their appearance, which affects their market value and the choice of the consumers. The manual sorting and inspection method takes a long time and is more tedious and exhaustive. Hence, an automated system is required to evaluate fruit, detect defects, and sort them based on their quality. Deep learning algorithms have highly influenced the area of object detection. Mask R-CNN and YOLOv5 are two object detection algorithms that have been experimented. YOLOv5 outperforms the Mask R-CNN approach when real-time object detection is required. The fruit identification and quality detection model is developed based on the YOLOv5 object detection system in the proposed work. The dataset includes 10,545 images of four different fruits, i.e., apple, banana, orange, and tomato, based on their quality. The model works in two phases. In phase 1, fruit is identified, and in phase 2, fruit quality detection is performed. The mosaic augmentation on the dataset has been applied for phase 1 training resulting in high detection performance and a robust system. The model classifies the fruit, and then the predicted image is passed to phase 2 for corresponding fruit quality detection. The mAP value of phase 1 is 92.80%. For phase 2, the mAP values for apple and banana quality detection models are 99.60% and 93.1%, respectively. The mAP values are 96.70% and 95% for orange and tomato quality detection models. The results show that the proposed method could realize fruit identification and quality detection on the validation dataset. The samples have been passed to show the real-time performance of the system. The efficiency of the trained model has been validated in different scenarios, including simple, complex, low-quality camera inputs. The fruit identification and quality detection model has been compared with several state-of-the-art detection methods, and the results are very encouraging.