Statistical model-based video segmentation and its application to very low bit-rate video coding

This paper presents a statistical model-based video segmentation algorithm for typical videophone and videoconference applications. This algorithm makes use of online information to build and track statistical models for both the background and foreground on the fly. The segmentation algorithm is then rendered as a MAP problem. A hierarchical system structure is designed and spatial and temporal filters are used to improve the segmentation quality. The algorithm is implemented on a PC and runs in real time. In addition, two possible applications are discussed: generating video objects for the upcoming MPEG-4 standard and introducing subjective factors into the rate control of DCT-based coding algorithms. We focus on the second application by proposing a rate-distortion (R-D)-based optimal rate control algorithm for H.263. In this rate control algorithm, which we refer to as region-based rate control algorithm, previous segmentation results are used as subjective knowledge. A distortion model is created to integrate both subjective and objective factors and an R-D criterion is used to obtain the optimal bit allocation. With the proposed algorithm, an H.263 compatible encoder is implemented and it produces better perceptual quality in our experiments than standard H.263. (C) 2000 Elsevier Science B.V. All rights reserved.