As one of important components of turbine-based thermoelectric conversion system, the flexible coupling has a significant effect on the dynamic characteristics and reliability of rotor. For the structural optimization problem of flexible coupling applied in the closed cycle power system, the natural frequency and critical speed of rotor is calculated, the orthogonal test of flexible coupling is designed, and the mathematical relationship between the natural frequency of rotor and the dimension parameters of flexible coupling is built. Then, the rules that the natural frequency of rotor change with the dimension parameters of flexible coupling are studied, and the sensitivity analysis of the dimension parameters of flexible coupling is done. The results show that the rotor has to pass three critical speeds with bending mode in the starting process. Because the critical speed corresponding to the third order bending mode is the highest, and correspondingly, the unbalance response of rotor is maximal, the third order bending mode is the most fatal to the rotor. Among the dimension parameters of flexible coupling, the diameter of flexible coupling shaft has the greatest impact on the natural frequency of rotor corresponding to the third order bending mode. The natural frequency of rotor corresponding to the third order bending mode decreases with the increase of length of flexible coupling shaft, and it increases with the increase of the diameter of shaft and the angle of elastic diaphragms near the electric motor and the engine. Finally, in order to reduce the unbalance response of rotor in the starting process, the structural parameters of flexible coupling are optimized. © 2022, Editorial Department of Journal of Propulsion Technology. All right reserved.