A Study on Image Reconstruction Based on Decoding fMRI Through Extracting Image Depth Features

Visualizing perceived content through functional magnetic resonance imaging (fMRI) analysis is a captivating research area in brain decoding. Previous studies have primarily focused on restoring either high-level semantic features or low-level semantic features from fMRI data, but rarely achieved effective restoration of both. This study proposes a novel approach for decoding the visual cortex activity measured by fMRI into the layered visual features that share the hierarchical information from the corresponding images. By iteratively optimizing the relationship between the layered visual features and the image's depth features extracted by a visual transformer, the method in this research significantly improves the reconstruction of the image's deep features. Furthermore, by incorporating the prior natural image information through a deep generator network, this work enhances the reconstruction process, resulting in richer semantic details. Experimental results verify the effectiveness of our methodology in restoring both high-level and low-level semantic features of the perceived images, ultimately enhancing the overall visual fidelity of the reconstructed image. Importantly, our model demonstrates successful generalization to reconstruct artificial shapes, indicating that the performance of our model is not simply achieved by relying on extensive sample datasets. These findings prove the efficacy of the method in effectively reconstructing the perceived content based on the hierarchical neural representations, providing a new method to study the brain's underlying mechanisms.