To ensure the smooth operation of each joint and shorten the joint movement time of a rail inspection robot, a trajectory planning method based on time optimization with a penalty function is proposed. According to the Denavit-Hartenberg (D-H) model of the inspection robot, a kinematic solution is found, and the trajectory of each joint is generated using a mixed polynomial interpolation algorithm. Taking time optimization as the standard, the traditional particle swarm algorithm cannot handle complex constraints, easily falls to local optimum solutions, and has a slow convergence speed. An improved simulated annealing particle swarm algorithm with a penalty function (IPF-SA-PSO) is proposed to optimize the trajectory generated by the mixed polynomial interpolation algorithm. The simulation results show that the proposed algorithm, compared with the mixed polynomial interpolation method, can limit the angular velocity and reduce the running time of each manipulator joint. The two algorithms are experimentally verified based on a rail inspection robot, and the results show that after adopting the optimization algorithm, the angular velocity of each joint is within the angular velocity limit, the run time is shorter, and the operation is smoother, which indicates the effectiveness of the proposed algorithm. The proposed algorithm can optimize the robot running time, improve the smoothness, and be applied to the fields of the automatic tracking of abnormal targets and video acquisition.