In this paper, we investigate the impact of the transmitter finite extinction ratio and the receiver carrier recovery phase offset on the error performance of two optically preamplified hybrid M-ary pulse position modulation (PPM) systems with coherent detection. The first system, referred to as PB-mPPM, combines polarization division multiplexing (PDM) with binary phase-shift keying and M-ary PPM, and the other system, referred to as PQ-mPPM, combines PDM with quadrature phase-shift keying and M-ary PPM. We provide new expressions for the probability of bit error for PB-mPPM and PQ-mPPM under finite extinction ratios and phase offset. The extinction ratio study indicates that the coherent systems PB-mPPM and PQ-mPPM outperform the direct-detection ones. It also shows that at Pb=10-9\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$P_b=10^{-9}$$\end{document} PB-mPPM has a slight advantage over PQ-mPPM. For example, for a symbol size M=16\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$M=16$$\end{document} and extinction ratio r=30\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$r=30$$\end{document} dB, PB-mPPM requires 0.6 dB less SNR per bit than PQ-mPPM to achieve Pb=10-9\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$P_b=10^{-9}$$\end{document}. This investigation demonstrates that PB-mPPM is less complex and less sensitive to the variations of the offset angle θ\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\theta $$\end{document} than PQ-mPPM. For instance, for M=16\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$M=16$$\end{document}, r=30\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$r=30$$\end{document} dB, and θ=10∘\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\theta =10^{\circ }$$\end{document} PB-mPPM requires 1.6 dB less than PQ-mPPM to achieve Pb=10-9\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$P_b=10^{-9}$$\end{document}. However, PB-mPPM enhanced robustness to phase offset comes at the expense of a reduced bandwidth efficiency when compared to PQ-mPPM. For example, for M=2\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$M=2$$\end{document} its bandwidth efficiency is 60 % that of PQ-mPPM and ≈86%\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$\approx 86\,\%$$\end{document} for M=1024\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$M=1024$$\end{document}. For these reasons, PB-mPPM can be considered a reasonable design trade-off for M-ary PPM systems.