This paper investigates the transmission performance of a 1.55-Tb/s Nyquist-wavelength division multiplexing superchannel along with a mixed line rate approach using 4x111 Gb/s polarization-multiplexed quadrature phase shift keying (PM-QPSK) and 5x222 Gb/s polarization-multiplexed quadrature amplitude modulation (PM-16QAM) formats. The system transmission performance with acceptable bit error rate of 4x 10(-3) is analyzed over standard single mode fiber (SSMF), large effective area fiber (LEAF), and pure silica core fiber (PSCF). It is observed that SSMF increases 45% reach over LEAF, whereas PSCF increases 66% to 84% reach over LEAF for 4.5 b/s/Hz spectral efficiency of the system. Also, spectral efficiency x distance product of 24,750 and 10,350 b/s/Hz.km are obtained for PM-QPSK and PM-16QAM, respectively, which is, to the best of our knowledge, better in comparison with the other existing literatures for single core channel investigations. (C) 2018 Society of Photo-Optical Instrumentation Engineers (SPIE).