VS-CAM: Vertex Semantic Class Activation Mapping to Interpret Vision Graph Neural Network

Graph convolutional neural network (GCN) has drawn increasing attention and attained good perfor-mance in various computer vision tasks, however, there is a lack of a clear interpretation of GCN's inner mechanism. For standard convolutional neural networks (CNNs), class activation mapping (CAM) meth-ods are commonly used to visualize the connection between CNN's decision and image region by gener-ating a heatmap. Nonetheless, such heatmap usually exhibits semantic-chaos when these CAMs are applied to GCN directly. In this paper, we proposed a novel visualization method particularly applicable to GCN, Vertex Semantic Class Activation Mapping (VS-CAM). VS-CAM includes two independent pipeli-nes to produce a set of semantic-probe maps and a semantic-base map, respectively. Semantic-probe maps are used to detect the semantic information from the semantic-base map to aggregate a semantic-aware heatmap. Qualitative results show that VS-CAM can obtain heatmaps where the high-lighted regions match the objects much more precisely than CNN-based CAM. The quantitative evalua-tion further demonstrates the superiority of VS-CAM.(c) 2023 Elsevier B.V. All rights reserved.