In this article, an automatic tracking process for unmanned aerial vehicle (UAV) system to follow the target with maneuverability is developed, and a vision-based control method is proposed. At first, the image-based visual servoing (IBVS) position control technology is applied without any prior target information for the algorithm practicability. Besides, consider the situation with unavailable angular velocity measurement, a thorough nonlinear attitude controller with an observer based on contraction is proposed. Finally, the whole UAV system stability is demonstrated to be uniformly ultimately bounded, and the bound can be arbitrarily small. Simulation and experiment results are presented to show effectiveness of the designed controller.