Error floor of pulse amplitude modulation with adaptive sampling in time-dispersive fading channels

We consider the error floor of coherently and differentially detected PAM (pulse amplitude modulation) in time-dispersive fading channels; PAM includes PSK (phase shift keying) and QAM as special cases. We introduce a new Zak-transform based method for computing the error floor. This method can be applied to arbitrary modulation formats and a very general class of time-dispersive channels. We prove that unfiltered BPSK exhibits no error floor when the sampling phase is chosen adaptively and (for coherent detection) the basis pulse satisfies a symmetry condition. For filtered BPSK there is an error floor, which depends on the filter bandwidth. For higher-order modulations, there is always an error floor; it can be attributed to I-Q crosstalk.