Background: Environment-Protection-Agency Reference Doses (EPA RfDs) for phthalate intakes are based on limited evidence, especially regarding low-dose male-reproductive toxicity. This study investigates the association between phthalate exposure and semen parameters and reproductive hormones in a general population with low phthalate exposure compared to the EPA RfDs. Methods: TheMARHCS (Male-Reproductive-Health-in-Chongqing-College-Students) cohort recruited 796 male students, who experienced a relocation of campuses and shifting environmental exposure. Urine, semen and blood before and after the relocation was collected and investigated for: (1) the associations between 13 urinary phthalate metabolites and 11 semen/hormone outcomes (five semen parameters including semen volume, sperm concentration, total sperm number, progressive motility, normal morphology) and six serum reproductive hormones including estradiol, follicle-stimulating hormone, luteinizing hormone, prolactin, progesterone, testosterone; (2) re-analysis of the metabolite-outcome associations in the subjects with estimated phthalate intakes below the RfDs; (3) a change in phthalate metabolites and change in semen/hormone outcomes after the relocation; (4) the association between these changes. Results: (1) All but two semen/hormone outcomes were associated with at least one phthalate metabolite, e.g., each quartile monoethyl phthalate was associated with a 5.3%, 5.7% and 2.6% decrease of sperm concentration, total sperm number and progressive motility respectively. (2) In the subjects with phthalate intakes below the RfDs, these metabolite-outcome associations remained significant. (3) All metabolites except mono(2-ethylhexyl) phthalate declined after relocation (P < 0.001 respectively); at the same time, semen volume, normal morphology, estradiol and luteinizing hormone increased (by 5.9%, 25.0%, 34.2% and 10.0%) and testosterone decreased (by 7.0%). (4) The changes in semen volume, normal morphology, estradiol and testosterone, but not the change in luteinizing hormone after relocation, were associated with the changes in the phthalate metabolites. Conclusions: Phthalate exposure is associated with interrupted semen quality and reproductive hormones in the human population even with a dose given below the RfDs. These effects, however, may only partially revert back when exposure decreases, thus emphasizing the urgency of stricter phthalate administration.