Mining activity is an important source of heavy metals in agricultural soils, threatening food safety and human health. In the present study, an integrated approach of Nemerow pollution index (PN), Geo-accumulation index (Igeo), potential ecological risk index (PERI), principal component analysis (PCA), cluster analysis (CA), and positive matrix factorization (PMF) was applied to comprehensively illustrate the pollution, potential ecological risk, and sources of heavy metals in agricultural soils surrounding a legacy Pb-Zn mine. The agricultural soils were seriously polluted and suffered high risks of heavy metals with the order of Cd > Pb > Hg > As > Zn > Cu > Ni > Cr. According to the PMF model, Cd (85.5%) and Zn (15.9%) originated from irrigation of polluted water. As (76.2%) and Pb (26.4%) were related to Pb/Zn ores transportation. Pb (52.5%), Zn (51.1%), and Cu (38.8%) were associated with atmospheric deposition during Pb-Zn mining. Hg (67.3%) was mainly from agricultural sources. Cr (59.5%), Ni (59%), and Cu (28.7%) came from the natural parent materials. Pb-Zn mining activity was the priority source accounting for 57.7% of heavy metals pollution in the agricultural soil. Additionally, agricultural and natural sources contributed 19.8% and 22.5%, respectively. These results provide valuable information for future prevention, remediation, and management of soil heavy metals pollution surrounding the Pb-Zn mining region.