We study the commuting problem for Toeplitz operators on the harmonic Bergman space of the unit disk. We show that an analytic Toeplitz operator and a co-analytic Toeplitz operator with certain noncyclicity hypothesis can commute only when one of their symbols is constant. We also obtain analogous results for semi-commutants.