Adaptive equalization of transmitter and receiver IQ skew by multi-layer linear and widely linear filters with deep unfolding

We propose a multi-layer cascaded filter architecture consisting of differently sized strictly linear (SL) and widely linear (WL) filters to compensate for the relevant linear impairments in optical fiber communications including in-phase/quadrature (IQ) skew in both transmitter and receiver by using deep unfolding. To control the filter coefficients adaptively, we adopt a gradient calculation with back propagation from machine learning with neural networks to minimize the magnitude of deviation of the filter outputs of the last layer from the desired state in a stochastic gradient descent (SGD) manner. We derive a filter coefficient update algorithm for multi-layer SL and WL multi-input multi-output finite-impulse response filters. The results of a transmission experiment on 32-Gbaud polarization-division multiplexed 64-quadrature amplitude modulation over a 100-km single-mode fiber span showed that the proposed multi-layer SL and WL filters with SGD control could compensate for IQ skew in both transmitter and receiver under the accumulation of chromatic dispersion, polarization rotation, and frequency offset of a local oscillator laser source. (C) 2020 Optical Society of America under the terms of the OSA Open Access Publishing Agreement