Glassy unsaturated polyester (UP) resin was reinforced using an organically modified montmorillonite (OMMT) and toughened with core-shell rubber (CSR) particles. The nanostructure, morphology, and deformation mechanism of composite specimens were studied by small-angle X-ray scattering (SAXS), transmission electron microscopy (TEM) and reflected optical microscopy (ROM). An intercalated nanostructure with partial exfoliation was observed in the UP reinforced by various amounts of OMMT. Locally clustered but globally good CSR particle dispersion in the UP matrix was evident in UP toughened with 5 and 10 wt% CSR particles. Simultaneous presence of OMMT and CSR particles in UP/OMMT/CSR hybrid composites was found to cause partial phase separation with bigger rubber particle agglomerates and lower clay-intergallery height increase. The effects of OMMT and CSR contents on volume shrinkage, impact fracture energy, fracture toughness, and compressive yield strength of UP were investigated. The introduction of OMMT of up to 3 wt% into the UP matrix lowered volume shrinkage to some extent, while further addition increased the shrinkage slightly. In the hybrid nanocomposites, the volume shrinkage decreased to a minimum level of 5.2 % with increases in OMMT level. The impact fracture energy of UP improved with increasing the OMMT level of up to 3 wt%, whereas its further addition decreased the impact fracture energy slightly due to the clay particle agglomeration. The hybrid composites with OMMT level below 3 wt% showed higher impact fracture energy compared to the reinforced UP specimens with the same OMMT levels. Interestingly, a synergism in the fracture toughness (K-IC) was observed in the hybrid composite containing 1 wt% OMMT and 10 wt% CSR particles. The presence of OMMT as reinforcement in the hybrid composites could compensate the lowering of the compressive yield strength caused by low-modulus CSR particles. The clay-rubber particle interaction in the hybrid systems seems to increase the threshold of shear deformation of the UP matrix to some extent.