A short note on deep contextual spatial and spectral information fusion for hyperspectral image processing: Case of pork belly properties prediction

This study demonstrates a new approach to process hyperspectral images where both the contextual spatial information as well as the spectral information are used to predict sample properties. The deep contextual spatial information is extracted using the deep feature extraction from pretrained resnet-18 deep learning architecture, while the spectral information was readily available as the average pixel values. To fuse the information in a complementary way, a multiblock modeling approach called sequential orthogonalized partial least squares was used. The sequential model guarantees that the information learned is complementary from spatial and spectral domains. The potential of the approach is demonstrated to predict several physical and chemical properties in pork bellies. The fusion of spatial and spectral information allowed better prediction of physical properties of pork samples. The spatial feature alone was able to predict both the chemical and physical properties. Deep features can be extracted for any type of images and can be fused with spectral data to enhance data modeling.