This paper proposed a high performance video coding system in redundant wavelet domain utilizing a multihypothesis variable size block motion compensation technique. The redundant wavelet transform (RDWT) provides several advantages over the traditional wavelet transform (DWT). The RDWT retains all the phase information of wavelet transform and provides multiple prediction possibilities in ME/MC in transform domain. The paper presents a new adaptive partitioning scheme and decision criteria that utilizes more effectively the motion content of a frame in terms of the size and shape of the blocks. The experimental results show that our proposed MH-VSBMC has a better performance and uses less number of partition blocks. In addition, the approach using selective strategy for subpixel accuracy achieves similar PSNR and other quality values while using less computation steps.