The unified PCHD modeling with inverter is established for the high performance of the speed control system of SPMSM. The passive controller of the SPMSM drive system is designed based on the interconnection and damping assignment of the energy-shaping method and port-controlled Hamiltonian systems with the dissipation(PCHD) theory. The speed regulator is designed by using the adaptive sliding mode method with extended state observer(ESO) in order to get desired q axis current and estimated speed, and the nonlinear disturbance caused by the inverter is compensated by using the extended state observer. The simulation results show that the proposed method realizes the veracity of speed estimation, so that the system has satisfactory dynamic and static performance.