The recently proposed space tether-sail system could enable various interesting and important potential space missions thanks to its thin and long connecting tether and solar radiation pressure (SRP) force without consuming propellants. This paper focuses on nonlinear libration and its control of a tether-sail system in Earth polar orbits. Firstly, nonlinear coupled in-plane and out-ofplane dynamics of the system with a rigid mass-distributed tether and two point masses (a chief satellite and sail respectively) is established using the Lagrangian equation method. Secondly, the predicted occurrence of chaotic in- and out-of-plane librational motions is verified utilizing numerical tools such as presenting the time history of libration, the phase plane, the Poincare<acute accent> section and power spectral density(PSD). Specifically, the paper investigates the sensitivity of nonlinear dynamical responses to initial values, focuses on the chaotic motion caused by the coupling between the in- and out-of-plane librations, the J2 perturbation of the Earth gravitational field and the small eccentricity of the orbits, also studies the nonlinear librational characteristics influenced by the initial mechanical energy (the Hamiltonian). Thirdly, chaotic librational motions will be controlled merely using modulatable SRP force by the sail propellantlessly. A sliding mode controller based on exponential approaching law is developed. To mitigate the effects of control torque saturation, an input compensator based on Radial Basis Function (RBF) neural network is designed. In addition, to address the cumbersome and inefficient manual tuning of control parameters for the sliding mode controller, a parameter tuning algorithm based on Learning Automata is designed. This algorithm is capable of tuning a highquality set of controller parameters within 100 iterations. The effectiveness of the propellantless actuators for chaotic libration motion control and developed control algorithm is supported by numerical simulations.
