An abnormal behavior detection algorithm based on Hidden Conditional Random Fields (HCRFs) is presented. HCRFs are introduced into video anomaly detection first time. We divide our model into three layers, video behavior feature layer, behavior semantic layer and anomaly behavior label layer. For the relationship of the patches' motion labels in time dimension, we propose three low-dimensional feature descriptors (PR, AIMB, IIDMB) as the inputs of video behavior feature layer. In the behavior semantic layer, we construct weights of the hidden state nodes to describe the distribution of the behavior features in space well by the relevant matching degree of patches' motion labels in the spatial dimension. Experiments show that our algorithm could achieve better detection results than other methods. Copyright © 2014 Binary Information Press.