Spatio-spectral feature classification combining 3D-convolutional neural networks with long short-term memory for motor movement/imagery

In this paper, we propose a novel EEG classification approach based on the Spatio-spectral feature, aiming to design a motor movement/imagery classification model that extracts multi-domain features with promising performance. Firstly, the difference between motor imagery (MI) tasks and real execution tasks was analyzed by calculating complexity measures of different frequency bands. Then, different connection patterns between MI tasks and real execution tasks were investigated by constructing the PLV and PLI networks. The results of the PLV and PLI brain networks showed that real execution tasks' network connections intensity was stronger than MI tasks, which meant these two networks could be used to distinguish the EEG signal features of different tasks. Afterward, to fully explain the multi-domain features of EEG signals, we fused the Phase Locking Value (PLV) and Phase-Lag Index (PLI) matrices (spatial-domain) under the subsets of frequency bands (frequency -domain) into a 3-D feature, namely the Spatio-spectral feature. Finally, a 3-D convolutional neural network combined with a long short-term memory (3DCNN-LSTM) was utilized to decode the feature. The results showed that the average accuracy of 10 subjects, 20 subjects, 50 subjects, 80 subjects, and 103 subjects was 85.88%, 83.09%, 76.30%, 75.02%, and 74.54%. Taken together, the proposed method provided promising classification accuracies, superior multi-domain features extraction ability, simpler structure, and robustness to classify the different motor movement/imagery tasks. The results contribute to our understanding of applying the deep learning method to decode EEG multi-domain features in the brain-computer interface (BCI) systems (e.g., MI, emotion recognition, and epileptic seizure classification).