Detecting the modality of a medical image using visual and textual features

Knowing the modality of a medical image is crucial in understanding the characteristics of the image. Therefore, it is important to classify medical images as per their modality. The image and its accompanying text caption contain information that could help in identifying the modality of a given medical image. This work proposes an approach for medical image modality classification using visual and textual features. The proposed approach uses convolutional neural networks to extract visual features from a medical image. Word embeddings obtained from biomedical word2vec models are used to generate textual features from the image captions. Support vector machine based classifiers are then used to classify medical images using these features. We propose to use the late fusion approach to combine visual and textual features. The proposed approach performs better than the state-of-the-art methods.