The self-excited vibration occurs due to mutual coupling between EMS maglev vehicles and the track. In order to study its effect on the safety and comfort of vehicles, a vehicle body-suspension frame-track dynamic model for single-magnet suspension systems was established. Using this model, the stability of the vehicle-track system was analyzed together with the influence of the suspension controller and main system parameters on the vibration. Consequently, a relational expression between the system parameters and stability was proposed, and a specific method of using instantaneous optimal control algorithm to inhibit the self-excited vibration of the vehicle-track system was discussed. The inhibiting effect on self-excited vibration of the proposed algorithm was simulated under three groups of different system parameters. The results indicate that the vehicle structure, the suspension controller, and the main parameters of the track would influence each other in the vehicle-track self-excited vibration. At the time of 10 s after the simulation system started, the amplitude of the suspension air gap vibration in the three parameter configurations decreased by 59%, 62%, and 5% respectively; and the amplitude of track vibration decreased by 48%, 94%, and 73% respectively, which demonstrated the effectiveness of the proposed control method. ©, 2015, Science Press. All right reserved.