Radiographs and texts fusion learning based deep networks for skeletal bone age assessment

Bone age assessment is a pediatric examination that determines the difference between skeletal age and chronological age. The discrepancy between the two ages will often trigger the likelihood of genetic disorders, hormonal complications and abnormalities of maturity in the skeletal system. Recently, although some automated bone age assessment methods by analyzing radiographs have been researched, the available text data from radiological reports are not used. Texts and radiographs are two different modals, the fusion of them can give us much more information for bone age assessment. In this paper, we present a novel multi-modal data fusion-learning network, called RT-FuseNet, for bone age assessment utilizing radiographs and texts. Specifically, we develop a convolutional neural network with spatial pyramid pooling layer and attention mechanism module to ensure the integrity of the image space information and enhance the subtle difference of features among radiographs respectively. In addition, texts are incorporated into the learning model to jointly learn non-linear correlations between various heterogeneous data. To evaluate the proposed approach, two datasets are used and several neural network structures are compared. Experimental results show that the proposed approach performs well.