Thermal detection of individual X-ray photons by small (0.5 x 0.5 mm) calorimeters has been used to achieve an energy resolution as good as 7.5 eV FWHM for 6 keV X-rays. Such detectors should have interesting applications in X-ray astronomy as well as laboratory spectroscopy, and they promise a high tolerance for embedded sources. Ideally, it should be possible to improve the resolution greatly by making smaller detectors or operating them at lower temperatures than the 50-100 mK currently used. However, there appear to be fairly fundamental limitations when semiconductor thermistors are used as the thermometer. When trying to achieve energy resolution of 0.1% or better, fluctuations in the thermalization efficiency of the detector must also be considered, and this places additional restrictions on suitable detector materials.