Nonlinear Precoding Design for MIMO Amplify-and-Forward Two-Way Relay Systems

In traditional multiple-input-multiple-output (MIMO) channel and MIMO one-way relay system, nonlinear precoding design has shown significant performance gain over linear design. In this paper, we aim to study nonlinear precoding design for MIMO amplify-and-forward (AF) two-way relay systems, where nonlinear minimal mean square error (MMSE) decision feedback equalizers (DFEs) are used in two destinations, and linear transmit precoding is applied at the source and relay nodes. We first investigate nonlinear precoding design, where the precoding is only conducted at two sources for a fixed-relay precoder. After some transformations, we prove that this design problem is convex, and an efficient algorithm is provided to find the optimal solution. Then, we consider the nonlinear joint precoding design to further incorporate relay precoding. Due to the nonconvexity of this problem, we first propose an iterative algorithm (Algorithm I) to approach the optimal solution. It is proven that Algorithm I is convergent and can converge to a stationary point of the joint design problem. Moreover, we present a simplified iterative algorithm (Algorithm II) for joint precoding design to reduce the design complexity. It is found that Algorithm II almost achieves the same performance as Algorithm I in most cases. Our simulation results show that the proposed nonlinear joint precoding design significantly outperforms the linear joint precoding design. It is also shown that the choice between the proposed nonlinear source precoding design and the linear relay precoding design is dependent on specific conditions.