We present the results of a comprehensive Spitzer survey of 70 radio galaxies across 1 < z < 5.2. Using IRAC (3.6 - 8.0 mu m), IRS (16 gm) and MIPS (24 - 160 mu m) imaging, we decompose the rest-frame optical to infrared spectral energy distributions into stellar, AGN, and dust components and determine the contribution of host galaxy stellar emission at rest-frame 1.6 mu m (H-band). The resultant stellar luminosities imply stellar masses of 10(11-12) M-circle dot, independent of redshift, indicating that radio galaxies form early and are amongst the most massive galaxies observed over this redshift range. These powerful radio galaxies tend to lie in a similar region of mid-IR color-color space as unobscured AGN, despite the inferred stellar contribution to their shorter-wavelength, mid-IR SEDs. The observed rest-frame mid-IR luminosities are typically above 10(11)L(circle dot), similar to lower redshift AGN. As expected, these exceptionally high mid-IR luminosities are consistent with an obscured, highly-accreting AGN. A weak, but significant, correlation of stellar mass with radio luminosity is found, consistent with earlier results.