Flux switching permanent magnet (FSPM) motors, especially outer-rotor FSPM (OR-FSPM), have caught our attention due to their advantages such as robust structure, torque density, energy density, and fault tolerance. However, the double salient pole structure of OR-FSPM motor causes large torque fluctuation, which will lead to the motor running unstably, poor working accuracy, and difficulty achieving good motor control. Therefore, this paper presents an optimization process for an OR-FSPM as motor/generator in flywheel energy storage systems (FESS). First, an initial 12/10 OR-FSPM is investigated through the finite element method (FEM). Second, the influences of single variables such as stator slot width, rotor tooth width, and air-gap length on the torque performance are studied. Third, a multi-variable optimization is processed through the response surface method (RSM) combined with FEM. Finally, the optimized 12/10 OR-FSPM is verified through FEM. The results of the initial structure and the optimized structure show that the average torque of the optimized motor is increased by 3.7 %, and the torque ripple is reduced by 36.06 %. In addition, the back-electromagnetic force (back-EMF) amplitude of the optimized motor is sharply reduced from 283.1 V to 192.8 V.