Decoding of multi-modal signals for motor imagery based on window positioning

With the rapid development of artificial intelligence, big data, and computing technology, multimodal research has gradually become an important research direction. This paper investigates the role of unimodal and multimodal data in improving recognition accuracy and overcoming inter-subject variability in Brain-Computer Interface systems based on Motor Imagery (MI). To address the challenge of data individuality caused by various factors, this paper focuses on the binary classification of left and right-hand MI tasks, utilizing a multimodal dataset (electroencephalogram, EEG, and functional near-infrared spectroscopy, fNIRS) for analysis. By applying a sliding time window technique to segment data, Common Spatial Pattern (CSP) is used to extract features from EEG data, while for fNIRS data, including oxyhemoglobin (HBO) and deoxyhemoglobin (HBR), statistical features such as mean and variance are extracted to capture their temporal characteristics. After feature-level fusion, stepwise regression is employed for feature selection, ensuring the refinement and efficiency of model input. The dataset is divided into a 90% training set and a 10% independent validation set. The training set undergoes five-fold cross-validation to ensure model reliability, and the independent validation set is used to assess model performance. Through training, the optimal time window is determined and validated on the independent validation set. In the five-fold cross-validation, the recognition accuracies for EEG, HBO, HBR, EEG + HBO, EEG + HBR, and the combination of all three data types are 91.13%, 81.83%, 87.39%, 94.99%, 96.52%, and 99.23%, respectively. In the independent validation, the recognition accuracies are 87.59%, 76.55%, 80.00%, 94.14%, 94.83%, and 98.28%, respectively. The difference in accuracy between five-fold cross-validation and independent validation is less than 2% for multimodal data and more than 3% for unimodal data. These results demonstrate that multimodal data can significantly enhance the decoding accuracy of brain signals during MI, improve robustness, and enhance generalization ability.