In this study, triaxial compression tests were conducted on coal to better understand the mechanism of coal burst disasters in deep coal mining. The mechanical properties, energy evolution and brittleness of coal under various true triaxial stress states were investigated. The evolution equation of the dissipated energy based on the logistic function was constructed, and the dissipated energy was defined as the damage factor D to determine the three-dimensional damage constitutive model of coal considering the residual strength. The findings indicate that, in the presence of true triaxial stress, the coal’s peak strength, residual strength, and elastic modulus increase as confining pressure rises while the absolute post-peak modulus drops. When confining pressure is low, coal failure manifests as brittle tensile–shear composite fracture; as confining pressure rises, the brittle characteristics gradually weaken, and shear fracture takes over as the primary failure mechanism. The brittleness index, which entirely considers the pre-peak and post-peak behaviors, is a better indication of the brittleness of coal under true triaxial conditions than other indices. The established damage constitutive model can adequately predict the coal’s strength and better characterize the coal’s complete stress–strain behavior, especially for high brittle coal.