Joint Transceiver Design for Secure Downlink Communications Over an Amplify-and-Forward MIMO Relay

This paper addresses joint transceiver design for secure downlink communications over a multiple-input multiple-output relay system in the presence of multiple legitimate users and malicious eavesdroppers. Specifically, we jointly optimize the base station (BS) beamforming matrix, the relay station (RS) amplify-and-forward transformation matrix, and the covariance matrix of artificial noise, so as to maximize the system worst-case secrecy rate in the presence of the colluding eavesdroppers under power constraints at the BS and the RS, as well as quality of service constraints for the legitimate users. This problem is very challenging due to the highly coupled design variables in the objective function and constraints. By adopting a series of transformation, we first derive an equivalent problem that is more tractable than the original one. Then, we propose and fully develop a novel algorithm based on the penalty concave-convex procedure (penalty-CCCP) to solve the equivalent problem, where the difficult coupled constraint is penalized into the objective and the resulting nonconvex problem is solved at each iteration by resorting to the CCCP method. It is shown that the proposed joint transceiver design algorithm converges to a stationary solution of the original problem. Finally, our simulation results reveal that the proposed algorithm achieves better performance than other recently proposed transceiver designs.