Recognition and counting of fruits playAa vital role in harvest estimation, harvesting, categorizing good and bad fruits, cost estimation, and stock estimation in departmental stores. Nowadays deep learning algorithms playAa major role in automatic object detection. Automating such mechanical robots faces challenges due to less accurate predictions because of background foliage, illuminations, and nightmares. In this computer vision task to detect and classify the intended objects, we designed and developed a Lightweight Self Attention Network (LwSANet) model. To reduce the amount of processing, and increase the object detection speed and performance, the Self-Attention Network Block was also introduced. LwSANet is simple to adopt and has obtained an accuracy of 99.25% and a loss of 0.003% for single fruit detection and classification. It has obtained an accuracy of 98.2% and a loss of 0.23% for the detection and classification of fruits from multiple and overlapped fruit images. When we compare with other state-of-the-art models the achieved accuracy is 1.68% better than other models. Further, the model performance is compared with various well-structured state-of-the-art architectures like LeNet, VGG-16, GoogLeNet, MobileNet, SqueezeNet, and ShuffleNet.