In order to explore the influence of the traction motor layout mode on the dynamic characteristics of rack vehicles and some nonlinear dynamic behaviors, and to propose the optimal layout of the traction motor of rack vehicles, the three-dimensional (3D) dynamic model of rack vehicle is established for the first time. The model considers the gear/rack nonlinear meshing behavior and the wheel/rail nonlinear contact relationship. The calculation method of time-varying dynamic excitation force is given. The motor layout mode of rack vehicle can be divided into front-mounted (FM) mode, rear-mounted (RM) mode, and dual-mounted (DM) mode. Adopting the model, the vehicle nonlinear dynamic characteristics under different traction motor layout modes are investigated, and the impact of three different driving forms on the vehicle is also studied. Results show that the gear/rack meshing characteristics of the vehicle in the DM mode are the best. The nonlinear motion of the gear system is quasi-periodic, and the stability is better when climbing. The vehicle in the DM mode is the smoothest in the straight section. Nevertheless, the nonlinear critical speed of rack vehicle is greatly reduced and it is easier to lose stability. The vehicle in the DM mode is the smoothest in the straight section. When climbing, the running comfort in DM mode is better than that in RM and FM mode. The excellent rate of vehicle running stability in the DM mode is 66.7%, which is 44.5% higher than that in FM mode and is 55.6% than that in RM mode. With the same total driving power, the gear meshing force in FM mode and RM mode is about twice the DM mode. Based on the calculation results in this paper, it is recommended that the optimal layout mode is DM mode.