Electroencephalogram (EEG) has been widely used in neurological disease detection, i.e., major depressive disorder (MDD). Recently, some deep EEG-based MDD detection attempts have been proposed and achieved promising performance. These works, however, still suffer from the following limitations, such as insufficient exploration of the EEG-based topological structure, information loss caused by high-dimensional data compression, and under-estimation of intra-class difference and inter-class similarity. To solve these issues, we propose an EEG-based MDD detection model named Self-attention Graph Pooling with Soft Label (SGP-SL). Specifically, we explore the local and global connections among EEG channels to construct an EEG-based graph in advance. By leveraging multiple self-attention graph pooling modules, the constructed graph is then gradually refined, followed by graph pooling, to aggregate information from less-important nodes to more-important ones. In this way, the feature representation with better discriminability can be learned from EEG signals. In addition, the soft label strategy is also adopted to build the loss function, aiming to further enhance the feature discriminability. Experimental results on the MODMA dataset demonstrate the superiority of the proposed method. What's more, extensive ablation studies are conducted to verify the effectiveness of the proposed elements in our model.