Transceiver Optimization for Two-Hop MIMO Relay Systems With Direct Link and MSE Constraints

In this paper, we investigate the transceiver design in a two-hop amplify-and-forward (AF) multiple-input multiple-output (MIMO) relay system with the direct link and quality-of-service (QoS) constraint. The QoS criterion is specified by the upper bound of the mean-squared error of the signal waveform estimation at the destination node. To minimize the total system transmission power, we apply a new AF relay protocol where the source node transmits signals during both time slots. Two iterative algorithms are proposed to jointly optimize the source and relay precoding matrices for the general case with multiple concurrent data streams, and the special case where a single data stream is transmitted, respectively. Simulation results show the effectiveness of both proposed algorithms. Interestingly, for the single data stream case, the second algorithm converges faster than the first algorithm. Compared with conventional two-hop MIMO relay systems where the source node is silent at the second time slot, the new protocol reduces the system transmission power. Under a given system transmission power, the proposed algorithms yield a higher system mutual information thanks to a better utilization of the direct link. Simulation results in this paper shed lights on some fundamental questions in relay system design such as when direct communication between source and destination (without any relay nodes) is optimal, and when the new AF relay protocol provides a larger gain compared with the conventional single phase source power allocation.