Orthant-symmetric four-dimensional geometric shaping for fiber-optic channels via a nonlinear interference model

Maximizing the data throughput for optical fiber communication via signal shaping has usually been regarded as challenging due to the nonlinear interference and implementa-tion/optimization complexity. To overcome these challenges, in this paper, we propose an efficient four-dimensional (4D) geometric shaping (GS) approach to design 4D 512-ary and 1024-ary modulation formats by maximizing the generalized mutual information (GMI) using a 4D nonlinear interference (NLI) model, which makes these modulation formats more nonlinear-tolerant. In addition, we propose and evaluate a fast and low-complexity orthant-symmetry based modulation optimization algorithm via neural networks, which allows to improve the optimization speed and GMI performance for both linear and nonlinear fiber transmission systems. The optimized modulation formats with spectral efficiencies of 9 and 10 bit/4D-sym demonstrate a GMI improvement of up to 1.35 dB compared with their quadrature amplitude modulation (QAM) counterparts in additive white Gaussian noise (AWGN) channel. Numerical simulations of optical transmission over two types of fibers show that the 4D NLI model-learned modulation formats could extend the transmission reach by up to 34% and 12% with respect to the QAM formats and the AWGN-learned 4D modulation formats, respectively. Results of effective signal-to-noise ratio are also presented, which confirm that the extra gains in optical fiber channel come from the enhanced SNR by reducing the modulation-dependent NLI.