Soil microbial diversity and community structure play pivotal roles in maintaining multiple ecosystem functions and biogeochemical cycling. However, the patterns and mechanisms of grazing effects on bacterial and fungal diversity remain poorly understood on a broad scale. To address this issue, we conducted a meta-analysis by compiling a comprehensive dataset consisting of 543 paired observations from 48 manipulative grazing experiments in grassland ecosystems. Pooling all data, our results indicated that grazing significantly altered soil bacterial and fungal community structure, almost uniformly decreased beta diversity, and did not change alpha diversity. However, grazing intensity had different effects on bacterial and fungal communities, with more pronounced alterations in the bacterial community. Moreover, bacterial alpha diversity showed a positive response to cattle/yak grazing and short-term grazing (e.g., less than 5 years), while sheep grazing and long-term grazing had negative effects on bacterial and fungal alpha diversity. Model-selection analysis further suggested that changes in soil organic carbon (SOC), total nitrogen (TN), and soil pH, grazing duration, and livestock type jointly regulated the responses of bacterial and fungal diversity and community structure to grazing disturbances. Our findings underscore the divergent responses of soil microbial alpha and beta diversity to grazing in grassland ecosystems. Appropriate livestock selection (e.g., cattle), short grazing duration, and moderate intensity can help to maintain the stability of microbial communities and ecosystem functions. These insights are crucial for establishing precision grazing management strategies, promoting the sustainability of grassland ecosystems.