A transformer based real-time photo captioning framework for visually impaired people with visual attention

In recent years, transformer-based photo captioning frameworks plays a crucial role in improving individuals’ overall well-being, self-reliance, and inclusivity by giving them access to visual content via written and voiced explanations. This research investigates a two-layer transformer architecture to capture extensive relationships in visual data effectively. Using a two-layer transformer design, visual attention mechanism enables the model to autonomously identify relevant regions within a picture while generating a particular word within the anticipated caption. To get the features from the images, a pre-trained Inception V3 model is used, and the feature maps are extracted from the final convolutional block. The dataset used in this research was Flick8k, Flick30k, among bespoke datasets. The model was trained on a TPU platform and implemented on a Raspberry Pi 4B graphics processing unit. To assess the efficacy of the two-layer transformer model for photo captioning, a series of experiments were undertaken using benchmark datasets and with state-of-the-art models. The quality of the generated captions was evaluated using quantitative criteria such as BLEU, BLEUR, ROUGE and F1 score. Furthermore, a series of qualitative assessments were conducted, employing human evaluators, to measure the overall coherence, and fluency of the captions. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.