Mechanical transmissions with noncircular gears are of interest to researchers, inventors, and engineers due to the high compactness of such mechanisms, as well as the implementation of a wide range of transmission functions. The kinematics of a planetary transmission with internal engagement with oval gears, which allows the implementation of reciprocating rotational motion of the output shaft, are examined. A kinematic model of the mechanism was constructed, and the law of motion was determined in the form of an analogue of the speed and position function of the output link. An experimental setup of the proposed planetary transmission has been developed, on the basis of which the position functions for three variants of the mechanism with different kinematic parameters have been investigated. Statistical analysis of the measurement errors showed the adequacy of the constructed kinematic model, which allows using it in future dynamic, power studies and the design of machines based on the proposed planetary transmission.