Along with the constant improvement of China's economic development, the quality-oriented education reform is being greatly implemented, in which training the high-quality talents with an all-round moral, intellectual, and physical development is the goal. The development of the physical education can not only improve the students' physical fitness, but also steel the students' will and enhance the physical quality. However, the traditional teaching method is always used by colleges and universities for physical education, and then, the enthusiasm of student is not fully aroused; sport items are simple and the infrastructure is imperfect. As a result, the development of physical education is affected. The hierarchical teaching model can be implemented for teaching students according to their aptitude; exercises are provided in combination with different physical conditions of students, aiming to improve the teaching effectiveness. In this paper, the problems and countermeasures of physical education in colleges and universities are discussed in combination with the hierarchical teaching model.