Viscoelastic dissipation is long concerned to be one of the origins of kinetic friction. In this study, the viscoelastic contact of rough solids is investigated by combining the principle of Lee and Radok and a recently developed incremental contact model. Evolutions of the real contact area and the average interfacial separation are ob-tained for a loading-unloading circle. For standard linear solids with Gaussian random rough surfaces, the effects of the viscoelastic reduction ratio and the loading rate on the contact responses are discussed. The viscosity of contacting solids would lead to the hysteresis of the mechanical responses to external excitation. Based on the relation between the average interfacial separation and the normal load, the viscoelastic energy dissipation is calculated and the role of surface morphology is addressed. Generally, the dissipated energy is proportional to the root-mean-square height of the rough surface, but decays for surfaces with larger root-mean-square slopes. The results of the current study are helpful for studying the friction between solids.