High spatial resolution (HSR) remote sensing imagery provides abundant geometric and detailed information, which is important for classification. In order to make full use of the spatial contextual information, object-oriented classification and pairwise conditional random fields (CRFs) are widely used. However, the segmentation scale choice is a challenging problem in object-oriented classification, and the classification result of pairwise CRF always has an oversmooth appearance. In this paper, a hybrid object-oriented CRF classification framework for HSR imagery, namely, CRF + OO, is proposed to address these problems by integrating object-oriented classification and CRF classification. In CRF + OO, a probabilistic pixel classification is first performed, and then, the classification results of two CRF models with different potential functions are used to obtain the segmentation map by a connected-component labeling algorithm. As a result, an object-level classification fusion scheme can be used, which integrates the object-oriented classifications using a majority voting strategy at the object level to obtain the final classification result. The experimental results using two multispectral HSR images (QuickBird and IKONOS) and a hyperspectral HSR image (HYDICE) demonstrate that the proposed classification framework has a competitive quantitative and qualitative performance for HSR image classification when compared with other state-of-the-art classification algorithms.