The effect of charge state modification at the Mn site on the physical properties of CMR manganites is reported. With a view to avoiding additional complexity of local spin coupling effects, Mn site substitution of La0.67Ca0.33MnO3 is carried out with appropriate diamagnetic ions - Zn2+, Zr4+, Ta5+ and W6+- of different valence states. The substitution results in size changes of the unit cell and enhanced local structural distortions, which increase in the order Zn, Zr, Ta and W. The ground state is ferromagnetic - metallic below a certain critical concentration chi(c) of the substituents, beyond which the magnetic ground state shows a glassy behaviour. The phase transition temperatures ( T-MI and T-c) decrease with substitution, but to different extents. The observed suppression rates of the Curie temperature, T-c, of similar to 39 K/ at.% and similar to 45 K/ at.% respectively for Ta5+ and W6+ substituted compounds are the highest reported in the Mn site substituted CMR manganites. Besides the modification of majority carrier concentration due to the increased ( decreased) Mn3+ concentration and enhanced local structural effects, the local electrostatic potential of the substituents seems to contribute to the unusually strong reduction in the itinerant ferromagnetism and the observed glassy states.