Because of large number of discontinuities existing in rock mass, the mechanics and failure modes are controlled by the rock mass structure characteristics. The numerical test method, integrating the results of in-situ investigations, laboratory and in-site tests, can provide a new approach to study the mechanical behavior and failure theory of fractured rock mass. The fractured sandstone rock masses in the west area of Hubei Province were chosen as a case study. Based on the statistic analyses of discontinuities, the structure models of rock masses are founded with Monte-Carlo method. Then a series of numerical compress tests using Distinct Element Method were carried out to discuss the failure strengths and failure modes in rock masses. The study shows that, the failure strengths and failure modes have a strong dependency on model size, confining pressure and loading orientation. In addition, the fractured rock mass shows anisotropic behavior, but this behavior is insignificant with the increase of confining pressure and the side length of rectangle model. The failure modes in fractured rock masses can be generalized into two types. One occurs in a complex way in small models under high confining pressures, i. e., combining sliding and bursting of discontinuities and shear bands in the rock matrix. Another takes place with slide of discontinuities in small models under low confining pressures or in large models. However, the pathways of these two failure modes both have an angle of 30 degrees to 45 degrees to the loading orientation.