This work presents a combined experimental and crystal plasticity finite element modeling study on the development of bulk and local residual strains during tensile and cyclic deformation of an austenitic stainless steel. The ( hkl )- specific bulk (residual) lattice strains are measured using X-ray Diffraction, while the local residual strains are measured using High Resolution Electron Back Scatter Diffraction. The residual strains are predicted using a dislocation density- based crystal plasticity model, with consideration for directional hardening due to backstress evolution. The work emphasizes on residual strain developments for four specific grain families: (111), (001), (101) and (311), specifically in terms of their correlation with the underlying microstructure, studied using crystallographic orientation, misorientation, dislocation density and backstress evolution. Large intragranular orientation gradients, dislocation densities and backstress are observed during tensile deformation for the texturally dominant (101) grain family, indicating that these grains have higher plastic deformation as compared to the (001) and (111) grain families. This also contributes to the observed relaxation in lattice strains for the (101) grain family, with the resulting load shed being primarily accommodated by the (001) grain family. In contrast, no such orientation gradients or lattice strain relaxations are observed in the cyclically deformed material. The measured local residual strains, which are also qualitatively predicted by the crystal plasticity simulations, highlight the additional effect of spatial heterogeneity and neighboring grains on the development of residual strains. Finally, statistical analysis of the simulated residual strains reveals that the hierarchy in the development of lattice strains is in the following order for the different grain families: (001) > (311) > (111) > (101) for tensile deformation, and (001) > (311) > (111) similar to (101) for cyclic deformation. The dominant factors contributing to the observed hierarchy are the elastic stiffness and the grain rotations (or lack thereof) for the different grain families during tensile and cyclic deformation.