Novel features in stretched-pulse and similariton mode-locked regimes of Yb-doped fiber laser with photonic bandgap fiber used for dispersion compensation are found by means of numerical simulations. We show that the mode-locked pulse may become shorter with increasing third-order dispersion. Analytical estimations explain observed behavior through resonant interaction of the main pulse with dispersive waves involving both resonant sidebands and zero-group-velocity dispersion waves. Switching between the stretched-pulse and the similariton regimes is also studied. (c) 2007 Optical Society of America.