The reduced order model of the system and related oscillation frequency are identified by total least squares-estimation on signal parameters via rotational technique (TLS-ESPRIT) algorithm, and then using the linear matrix inequality (LMI) based robust control design method a multi-channel additional controller for different oscillation modes occurred in HVDC power transmission system is designed. The balanced truncation method is used to reduce the order of the controller, and both performance optimization and robustness of the controller are taken into account, meanwhile low-frequency oscillation and sub-synchronous oscillation can be suppressed simultaneously. The interaction between control channels is restrained by Butterworth filter in each control channel. To validate the performance of the designed controller, a traditional PID controller is design as the contrast. Simulation results show that the designed multi-channel robust controller can well suppress different oscillation modes simultaneously and possesses strong robustness. Besides, to be convenient for practical application, output feedback control is utilized in the design of the controller.