Pneumonia is a leading cause of death worldwide, particularly in Pakistan. Chest X-ray (CXR) images serve as a primary means of pneumonia detection. However, manual interpretation by radiologists is challenging, necessitating the development of automatic computer-aided diagnostic systems to enhance accuracy. Recent literature has explored diverse deep learning algorithms for pneumonia detection, but their practicality is limited by high computational demands and the need for fast GPUs. In this study, a lightweight approach leveraging transfer learning of pre-trained architectures (SSD MobileNet V2, SSD MobileNet V2 FPNLite 320x320, and SSD MobileNet V2 FPNLite 640x640) were employed, followed by comparative analysis of these pre-trained models for pneumonia detection. The proposed models underwent extensive experiments in various scenarios, employing different dataset distributions, hyper-parameters, classification loss functions, and image pre-processing techniques using a set of evaluation metrics. The SSD MobileNetV2, SSD MobileNetV2 FPNLite 320x320, and SSD MobileNetV2 FPNLite 640x640 models achieved mAP scores of 76%, 85%, and 80%, respectively, alongside accuracies of 81.3%, 94.6%, and 92.6% on an unseen dataset from the Guangzhou Women and Children's Medical Center pneumonia dataset. Notably, the SSD MobileNetV2 FPNLite 320x320 model exhibited superior performance among the three. These findings demonstrate great potential in accurately detecting pneumonia cases in medical images, offering computational efficiency, cost-effectiveness, and faster results compared to existing methods in the literature.