Crack image classification and information extraction in steel bridges using multimodal large language models

Existing deep learning methods fail to meet the requirements of zero-shot learning scenarios for crack detection and have yet to investigate the specific impact of visual prompts on the detection performance of multimodal large language models (MLLMs). This paper proposes a cascaded crack detection strategy based on MLLMs, decomposing the crack detection task into a stepwise classification process from the image level to patch level. The crack detection performance of five MLLMs and five traditional deep-learning models was systematically evaluated, while the influence of visual prompt quantity and design on model performance was examined. The results indicate that MLLMs achieve performance comparable to deep learning models in image-level crack detection. However, in finer-grained patch-level crack detection, their performance still needs to catch up to that achieved by deep learning models based on Segmented Transformer. Increasing the number of visual prompts can partially improve the classification performance of MLLMs.