The Philippine lime (Citrofortunella microcarpa) locally known as Calamansi belongs to the citrus family and is among the top four produce of the Philippines exported to different countries. When done manually, postharvest grading of Calamansi is a laborious operation. Thus, an effective method of assessing them according to size, color, and maturity is required to assist calamansi farmers in grading and classifying it non-destructively with precision and accuracy. Deep learning architectures have been useful in a wide range of agricultural applications including classifying, grading fruits, and quality recognition. The goal of the study is to compare the performance of four deep learning architectures: VGG16, ResNet50, InceptionV3, and DenseNet201 in classifying Calamansi. Three types of optimizers were considered in the training including Adam, SGD, and RMSProp. Furthermore, data augmentation is applied to the datasets to reduce overfitting and increase variation in the training datasets because of limited data. With Adam as optimizer, classification accuracies resulted to 96.98%, 91.48%, 27.95% and 65.95% for ResNet50, VGG16, InceptionV3 and DenseNet201, respectively. Classification accuracies of 91.76% for ResNet50, 88.19% for VGG16, 25% for InceptionV3 and 39.84% for DenseNet201 where observed implementing the SGD optimizer. Lastly, 95.60%, 75.55%, 25% and 63.19% classification accuracies were obtained for ResNet50, VGG16, InceptionV3 and DensNet201, respectively when RMSProp was utilized as the optimizer. From the CNN architectures used, ResNet50 acquired the best result in classifying Calamansi. The model resulted in an overall accuracy of 96.98% utilizing Adam as optimizer. Based on the findings of the study, it was determined that Resnet50 architecture with Adam as optimizer may be utilized to categorize Calamansi fruit or the Philippine lime.