We give formulas for the extension groups between tautological sheaves and more generally between tautological objects twisted by natural line bundles on the Hilbert scheme of points on a smooth quasi-projective surface. As a consequence we observe that a tautological object can never be a spherical or P-n-object. We also provide a description of the Yoneda products.