In this paper, an error reduction method for the roller gear cam (RGC) mechanism is developed to replace conventional handwork method. A testing machine, capable of measuring kinematic and dynamic responses, was designed and implemented to experimentally verify the new method. By comparing the measured kinematic and dynamic responses with the theoretical data of a simplified model, characteristic error curves could quantitatively determine the mating state between the turret rollers and the cam ribs. The qualitative method, by identifying error shapes, was used to locate error sources. In addition, the slope comparison method was used to determine the direction and quantity of the errors. To compensate for the errors found, an adjustment criteria was designed by setting two adjustable variables of eccentric plates to fine-tune three assembly parameters. Based on the experimental approach, this error reduction method can practically improve the assembly quality, and achieve possible optimum performances of the mechanism with manufacturing errors.