Heterogeneous Graph-Based Finger Trimodal Fusion

Multimodal graph fusion utilizes the modeling of weighted graph to combine information from multiple sources in order to improve recognition performance whilst addressing some limitations of traditional methods. In this work, we propose a novel approach for the finger trimodal graph fusion, by extending the crystal-like bijective structure to the Atomic Group-like heterostructure. The crucial aspect is to establish the inter-modal correlation. Inspired by chemical bonds, we construct atom serial numbers by pre-clustering the divided image blocks in one modality. Then, we trained a node classifier to share the set of serial numbers among the three modalities. As a result, nodes across modalities can be bonded. Furthermore, both the feature attributes and position information are taken into consideration for the intra-modal edges of the fused graph. Finally, we design heterogeneous graphs with higher recognition potential. Experimental results show that with our strategy, we can improve matching accuracy to 96.6% while lowering the equal error rate to 2.687%.