In this paper, we have designed an ultra-dense wavelength division multiplexing system for 450 channels with a bit rate of 160 Gb/s per channel based on the high spectral efficient differential quadrature phase shift keying (DQPSK) modulation technique and investigated the performance of the system using different pulse shapes such as nonreturn-to zero, return-to-zero, carrier suppressed return-to-zero (CSRZ), modified duobinary return-to-zero and manchester, under the effect of various transmission impairments such as amplified spontaneous emission, group velocity dispersion, self-phase modulation and cross-phase modulation. Bit error rate (BER) and quality factor (Q-factor) have analyzed with variation in transmission distance and input power. It has also observed from the simulation results that the CSRZ pulse scheme serves the best performance in terms of BER and Q-factor due to its high tolerance towards linear and nonlinear effects. The maximum Q-factor of 14 dB and minimum BER of around 10(- 13) have reported for CSRZ for a distance from 50 to 400 km. In the end, the eye diagrams for all the pulse shapes have compared under the influence of the combined effect of the above mentioned transmission impairments and found better eye-opening in CSRZ-DQPSK modulated system as compared to others.