This paper presents an analytical examination of the small-single stability (SSS) criterion of the permanent magnet synchronous generator (PMSG)-based wind power delivery system via voltage source converter-based high voltage direct current (VSC-HVDC). First, a small-signal model of the PMSG-based WPDS is developed. Then, the SSS criterion, driven by the phase-locked loop of the PMSG in the sub-synchronous timescale, is derived. The derived SSS criterion provides analytical insights into why and how the loading condition, the grid connection, and the control parameters affect the system's SSS. It is unambiguously revealed that increasing loading conditions of the PMSG or/and the grid connection of the WPDS to VSC-HVDC shall bring about a higher risk of oscillatory instability. Hence, analytical derivation of the SSS criterion helps better understand the instability mechanism in the PMSG-based WPDS via VSC-HVDC. In addition, while the derivation of the SSS criterion presupposes identical dynamics among PMSGs, this derived criterion can still be approximately utilized to assess the SSS of the PMSG-based WPDS via VSC-HVDC, irrespective of whether the dynamics of the PMSGs are similar or different. Finally, the SSS criterion is demonstrated and evaluated through three examples of the PMSG-based WPDS via VSC-HVDC.