To detect moving pedestrians in video surveillance rapidly and accurately, the pedestrian detection process is divided into two stages: motion detection stage and classification stage. In the motion detection stage motion detection is exploited to rapidly get the motion regions. By getting these interesting regions which may contain pedestrians, the regions used for next stage are greatly reduced and the whole pedestrian detection speed is improved. In the classification stage HOG features are used to classify interesting regions into pedestrian region and non-pedestrian region. In the motion detection stage, an improved frame differential method is used. The threshold changes with the pixel value changes and can adapt to the environment change, such as background difference, lightness change and motion speed. In the classification stage, an improved HOG method is used. Multi-scale HOG features are used to get better performance and Adaboost algorithm is used to train cascaded classifier for a higher detection speed. To evaluate the performance of our method, we test it on several video sequences taken from different scenes. Experiment results show that our method achieves good performance with an average over 90% detection rate. And our method can process over 20 frames per second and satisfies the demand for real-time task.