Irregular lattice models are developed to simulate fracture of multiphase particulate materials, such as concrete. The models are composed of rigid-body-spring elements that break according to simple rules. A salient feature of the models is the use of Voronoi diagrams to define the lattice structure and assign the elastic and fracture properties of the elements. The material is discretized as a three-phase composite consisting of a matrix phase, coarse inclusions, and the matrix-inclusion interfacial zones. Aggregates are randomly positioned in the domain according to a target granulometric distribution. A procedure is outlined for the explicit representation of the surfaces of such heterogeneous features, including control over the thickness of the matrix-aggregate interfacial zones. Fracture simulations are conducted for notched, three-point bend specimens of concrete, where each phase is assigned locally brittle fracture properties. The simulation results show both pre- and post-peak behavior that agrees with experimental findings, at least in a qualitative sense. In particular, toughening mechanisms form through interaction of developing cracks with the evolving material structure. However, the post-peak toughness is largely underestimated due, in part, to the coarse discretization of the material and the lack of frictional effects in the model. For comparison, the same specimen is analyzed using a homogeneous material model and a cohesive crack approach, which lumps the various energy dissipation mechanisms active at finer scales into a cohesive traction versus separation law.