A three-dimensional theory of Compton scattering is described, which accounts for the effects the electron beam emittance and energy spread on the scattered x-ray brightness. The radiation scattered by an electron subjected to an arbitrary electromagnetic field in vacuum is derived in the linear regime, neglecting radiative corrections; it is found that each vacuum eigenmode gives rise to a single Doppler-shifted classical dipole excitation. This formalism is then applied to Compton scattering in a three-dimensional laser focus, and yields a complete description of the influence of the electron beam phase-space topology on the x-ray brightness; analytical expressions including the effects of emittance and energy spread are also obtained in the one-dimensional limit. With these results, the x-ray brightness generated by a 25 MeV electron beam is modeled, taking into account the beam emittance, energy spread, and the three-dimensional nature of the laser focus; its application to x-ray protein crystallography is outlined.