In modern intensive agricultural production, soil nutrients and soil-borne diseases of cultivated ginseng are crucial factors influencing ginseng growth. However, it remains unclear how the addition of phosphorus fertilizer affects the ginseng rhizosphere soil and soil nutrient structures. Pot experiments were carried out to investigate the impacts of phosphorus fertilizer addition on soil properties, microbial communities, and ginseng physiological characteristics, using the amended ginseng land soil as the experimental material and the amended ginseng land soil without phosphorus fertilizer application as the control group. The results demonstrated that phosphorus addition significantly promoted the relative abundance of beneficial microorganisms in the rhizosphere, including Ascomycota and beneficial bacteria in the phyla of Actinobacteriota, Proteobacteria, and Gemmatimonadota. Meanwhile, phosphorus addition inhibited the relative abundance of the harmful fungi Fusarium and Nectria, suppressing pathogens. Compared with the control group, phosphorus addition significantly increased the content of monomer saponins Rg1, Re, and Rb1 as well as total saponins, enhanced the activities of catalase (CAT) and superoxide dismutase (SOD), and increased the content of ginseng soluble sugars (SS) and soluble proteins (SP). Additionally, phosphorus addition significantly increased the content of soil available phosphorus and alkaline dissolved nitrogen, elevated the activities of soil cellulase (S-CL), amylase (S-AMY), sucrase (S-SC), and nitrate reductase (S-NR), and decreased the content of soil organic matter, urease (S-URE), and protease activities (S-PRO). The core microbial symbiotic network diagrams suggest that fungi have more complex functional roles and more microbial interactions in the rhizosphere ecosystem. In summary, the addition of phosphorus fertilizer can significantly improve the rhizosphere environment of cultivated ginseng and the growth characteristics of ginseng, and can promote the accumulation of secondary metabolites of ginseng by regulating the soil microenvironment, which provides a new idea to improve the yield and quality of ginseng.