An algorithm for video sequence analysis and image segmentation by using 3D scene model is presented. At the beginning of the sequence, the procedure uses two frames to acquire the depth map, and represent the scene as a 3-D wire-frame computed from the depth map. A linear algorithm with low complexity is used to recover the motion parameters, and update the 3-D scene description for each additional frame. The approach ia applied to solve the problem such as video sequence segmentation, object tracking, and video object plane (VOP) generation.