Hexagonal image segmentation on spatially resolved transcriptomics

Spatial transcriptomics is a rapidly evolving field that enables researchers to capture comprehensive molecular profiles while preserving information about the physical locations. One major challenge in this research area involves the identification of spatial domains, which are distinct regions characterized by unique gene expression patterns. However, current unsupervised methods have struggled to perform well in this regard due to the presence of high levels of noise and dropout events in spatial transcriptomic profiles. In this paper, we propose a novel hexagonal Convolutional Neural Network (hexCNN) for hexagonal image segmentation on spatially resolved transcriptomics. To address the problem of noise and dropout occurrences within spatial transcriptomics data, we first extend an unsupervised algorithm to a supervised learning method that can identify useful features and reduce noise hindrance. Then, inspired by the classical convolution in convolutional neural networks (CNNs), we designed a regular hexagonal convolution to compensate for the missing gene expression patterns from adjacent spots. We evaluated the performance of hexCNN by applying it to the DLPFC dataset. The results show that hexCNN achieves a classification accuracy of 86.8% and an average Rand index (ARI) of 77.1% (1.4% and 2.5% higher than those of GNNs). The results also demonstrate that hexCNN is capable of removing the noise caused by batch effect while preserving the biological signal differences.