Zero and few shot action recognition in videos with caption semantic and generative assist

This research introduces a trimodal approach that integrates image caption, image generation and action semantic to achieve zero-shot and few-shot action recognition. The modules used to implement the modalities are Image Action Captioning, Generated Image Similarity Analyzer, and Action Semantics Analyzer. The Image Action Captioning model, uses CLIP (Contrastive Language-Image Pretraining) to generate captions. The Generated Image Similarity Analyzer employs the Stable diffusion model to generate action images, while the Action Semantics Analyzer combines action recognition and text embedding. Results of the three modules are integrated using soft voting. This approach allows for a more comprehensive understanding of different domains, thereby enhancing the accuracy of Zero-Shot and Few-Shot Action Recognition. Compared to the state-of-the-art model, the Caption Semantic Generative Assist enhances the accuracy of Zero-Shot Action Recognition by +4.97% for UCF101 dataset and +5.32% for HMDB51 dataset. Compared to the state-of-the-art model, the Caption Semantic Generative Assist enhances the accuracy of Few-Shot Action Recognition by +2.47% for UCF101 dataset and +11.18% for HMDB51 dataset. The source code of this work will be made available at https://github.com/GayathriThriloka/CSGA.git © The Author(s), under exclusive licence to Bharati Vidyapeeth's Institute of Computer Applications and Management 2024.