Background Although stage I colorectal cancer has an excellent prognosis after complete surgical resection, disease recurrence still occurs. This study aimed to assess prognostic risk factors in this early stage of disease. Methods All non-neoadjuvantly treated stage I colon (CC) and rectal (RC) patients who underwent a surgical resection between 2004 and 2015 were identified. Clinicopathological differences and long-term oncological outcomes were compared. Results CC patients (n = 433) were older and had more pre-existing comorbidities. RC patients (n = 86) were associated with more T2 tumors, venous invasion, and higher rates of 30-day morbidity. In multivariate analysis, lymphatic invasion was found to be an independent predictor for disease recurrence (OR 4.57, P = 0.010) and worse disease-free survival (HR 4.26, P = 0.012). This was particularly true for distant recurrence, with eight times higher hazard ratios when lymphatic invasion was present (HR 8.02, P < 0.001). T2 tumors were at risk, though no significant association was found (OR 3.86, P = 0.051; HR 3.61, P = 0.065, respectively). Conclusions Lymphatic invasion was strongly associated with worse DFS, in particular distant recurrence. This subgroup of stage I patients might benefit from a more intensive follow-up and maybe should be considered for adjuvant therapy.