Exploring the Limits of VLMs: A Dataset for Evaluating Text-to-Video Generation

Vision language models (VLMs) integrate vision and text by learning from images and their textual descriptions for generating text from images and vice-versa. VLMs are used for various tasks such as image captioning, and visual question answering. As VLMs extend to video generation tasks, significant challenges arise. The generated videos often lack temporal consistency, and there exist issues with alignment between the generated video content and the input text. This work proposes a set of prompts for systematically evaluating VLMs on text-to-video generation focused on object coherence and temporal consistency. This dataset of prompt covers two categories: first, the complexity of prompts, and second, the number of objects and actions. The first category consists of four levels of prompt complexity: simple, mid-level, high-level, and unique and rare object prompts. The second category also consists of four levels: single object, single object in action, multiple objects, and multiple objects in action. Thus, the dataset is a combination of 16 different prompt scenarios where each scenarios has 10 prompts resulting in a dataset of 160 prompts. This work explores and evaluates the outputs of three models from the family of diffusion-based VLMs for the task of text to video generation. The videos generated by the models were assessed by five participants on a 0-5 Likert scale. VLMs under study were able to generate temporally consistent videos and proper object in video in only 33.63% and 39.43% of the total evaluation scenario prompts. Most of these prompts belong to the categories of single object and simple prompts. The performance of models drops with increase in objects, actions and prompt complexity. The scenarios of actions, with rare prompts are where all models perform poorly. Thus, reflecting on limitations of VLMs in generating videos. The proposed dataset can be extended to other video generation models to benchmark their performances on the basic aspects of consistency and alignment in videos.