Decoding natural image stimuli from fMRI data with a surface-based convolutional network

被引:0
|
作者
Gu, Zijin [1 ,2 ]
Jamison, Keith [3 ]
Kuceyeski, Amy [3 ]
Sabuncu, Mert [1 ,2 ,3 ]
机构
[1] Cornell Univ, Sch Elect & Comp Engn, New York, NY 14853 USA
[2] Cornell Tech, New York, NY 10044 USA
[3] Weill Cornell Med, Dept Radiol, New York, NY USA
来源
MEDICAL IMAGING WITH DEEP LEARNING, VOL 227 | 2023年 / 227卷
关键词
functional MRI; neural decoding; image reconstruction; RECONSTRUCTION; REPRESENTATION; ADAPTATION;
D O I
暂无
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (Instance-Conditioned GAN) to train another mapping from brain response to fine-grained image features using a variational approach (Cortex2Detail). Image reconstructions obtained by our proposed method achieve state-of-the-art semantic fidelity, while yielding good fine-grained similarity with the ground-truth stimulus. Our code is available on https://github.com/zijin-gu/meshconv-decoding.git.
引用
收藏
页码:107 / 118
页数:12
相关论文
共 50 条
  • [41] Classification of ADHD based on fMRI data with Graph Convolutional Networks
    Gong, Qi
    Zhu, Jun-Sa
    Jiao, Yun
    PROCEEDINGS OF 2023 4TH INTERNATIONAL SYMPOSIUM ON ARTIFICIAL INTELLIGENCE FOR MEDICINE SCIENCE, ISAIMS 2023, 2023, : 500 - 504
  • [42] Image Classification Based on Convolutional Neural Network
    Prassanna, P. Lakshmi
    Sandeep, S.
    Rao, Kantha
    Sasidhar, T.
    Lavanya, D. Ragava
    Deepthi, G.
    SriLakshmi, N. Vijaya
    Mounika, P.
    Govardhani, U.
    SUSTAINABLE COMMUNICATION NETWORKS AND APPLICATION, ICSCN 2021, 2022, 93 : 833 - 842
  • [43] Arbitrary surface data patching method based on geometric convolutional neural network
    Fan, Linyuan
    Ji, Dandan
    Lin, Peng
    NEURAL COMPUTING & APPLICATIONS, 2023, 35 (12): : 8763 - 8774
  • [44] Arbitrary surface data patching method based on geometric convolutional neural network
    Linyuan Fan
    Dandan Ji
    Peng Lin
    Neural Computing and Applications, 2023, 35 : 8763 - 8774
  • [45] Motor Intention Decoding from the Upper Limb by Graph Convolutional Network Based on Functional Connectivity
    Feng, Naishi
    Hu, Fo
    Wang, Hong
    Zhou, Bin
    INTERNATIONAL JOURNAL OF NEURAL SYSTEMS, 2021, 31 (12)
  • [46] Classification of fMRI Data using Support Vector Machine and Convolutional Neural Network
    Zafar, Raheel
    Malik, Aamir Saeed
    Shuaibu, Aliyu Nuhu
    Rehman, M. Javvad ur
    Dass, Sarat C.
    2017 IEEE INTERNATIONAL CONFERENCE ON SIGNAL AND IMAGE PROCESSING APPLICATIONS (ICSIPA), 2017, : 324 - 329
  • [47] Early prediction of dementia using fMRI data with a graph convolutional network approach
    Han, Shuning
    Sun, Zhe
    Zhao, Kanhao
    Duan, Feng
    Caiafa, Cesar F.
    Zhang, Yu
    Sole-Casals, Jordi
    JOURNAL OF NEURAL ENGINEERING, 2024, 21 (01)
  • [48] A Convolutional Neural Network for Transcoding Simultaneously Acquired EEG-fMRI Data
    Liu, Xueqing
    Sajda, Paul
    2019 9TH INTERNATIONAL IEEE/EMBS CONFERENCE ON NEURAL ENGINEERING (NER), 2019, : 477 - 482
  • [49] Evaluation of volume-based and surface-based brain image registration methods
    Klein, Arno
    Ghosh, Satrajit S.
    Avants, Brian
    Yeo, B. T. T.
    Fischl, Bruce
    Ardekani, Babak
    Gee, James C.
    Mann, J. J.
    Parsey, Ramin V.
    NEUROIMAGE, 2010, 51 (01) : 214 - 220
  • [50] Natural image classification based on multi-parameter optimization in deep convolutional neural network
    Wang L.
    Zhang Y.
    Xi R.
    Ling L.
    International Journal of Performability Engineering, 2019, 15 (09): : 2515 - 2521