To overcome the disadvantages that the traditional particle filters based on color histogram is susceptible to environmental interference and illumination variations, an improved particle filter algorithm was proposed. This article starts from improving the description ability of the target feature model. First, the histogram weighted function was optimized. Second, for the shortcoming of the color feature, a new color local entropy target observation model was constructed by mapping the target from color feature space to local entropy space. In addition, in order to make the model better adjust to environmental interference and target deformation, an adaptive updating strategy of the target model was designed and the number of particle was adjusted dynamically. Experimental results demonstrate that the proposed algorithm is effective.