Background/Objectives: Sex differences in nutrition-related determinants of mild cognitive impairment (MCI) exist among the elderly. This study aimed to explore sex-specific influencing factors of MCI. Methods: A case-control study was conducted in 2020 involving 1086 elderly people aged 55 years and above from four sites in Zhejiang Province, China. Data on demographics, cognitive assessment, depression scale, daily food intake, and physical examinations were collected. The assessment of plant-based diet patterns depended on an overall plant-based diet index (PDI), a healthful plant-based diet index (hPDI), and an unhealthful plant-based diet index (uPDI). Multivariate logistic regression models were employed to assess the determinants of MCI in females and males. Results: Among 571 females, 141 (24.7%) had MCI, and 126 (24.5%) had MCI among 514 male participants. In females, the multivariate analysis revealed that being unmarried/divorced/widowed (OR = 1.95, 95% CI: 1.10-3.45), having depression (OR = 6.06, 95% CI: 1.87-19.66), and having a uPDI score >= 55 (OR = 2.41, 95% CI: 1.50-3.89) were associated with a significantly elevated risk of MCI. Conversely, a cereal consumption of >= 300 g/d (OR = 0.32, 95% CI: 0.19-0.53) was linked to a significantly reduced risk. In males, vegetable consumption >= 150 g/d (OR = 0.39, 95% CI: 0.23-0.66), vegetable oil consumption >= 22 g/d (OR = 0.502, 95% CI: 0.307-0.820), and cereal consumption >= 300 g/d (OR = 0.44, 95% CI: 0.27-0.71) were associated with a lower MCI risk. Meanwhile, rural residence (OR = 1.90, 95% CI: 1.12-3.25) and advanced age, especially 75 years old and above (OR = 4.71, 95% CI: 2.44-9.12), were also risk factors in males. Notably, the Restricted Cubic Spline (RCS) model showed that females with a uPDI score < 55 had a lower prevalence of MCI, while those with a score >= 55 faced a higher risk. Conclusions: This study indicates potential sex disparities in the risk factors for MCI. Future research should prospectively establish causal relationships. Additionally, precise intervention strategies are urgently needed.