Nonlinear optimal control, a dynamic approach and one of the applications of calculus of variations, is widely used in aerospace applications because of the performance criticality required. In this paper, a new nonlinear optimal controller is proposed for time-varying nonlinear dynamics of spacecraft formation flying with periodic coefficients using State-Dependent Riccati Equation (SDRE) technique. Over a more classic LQR control approach, the SDRE control approach has advantages of better tracking response, robustness, and ability to capture time-varying nonlinear characteristics of the system. First, time-varying cubic approximation model of spacecraft relative motion is developed from the original spacecraft relative motion. Second, four different linear-like State-Dependent Coefficient (SDC) parameterized structures, carefully selected to ensure optimal relative motion trajectory tracking of the formation flying, are developed for the approximated nonlinear system dynamics using SDRE. Finally, the efficiency of the new nonlinear SDRE controller and its optimal performance are validated using numerical analysis. © 2021, Shanghai Jiao Tong University.