In this paper, we study a geometric optimization problem of distributed multi-input multi-output (MIMO) radar system. We aim to enhance both the system surveillance and localization performance by adjusting the node positions. Different from existing researches, from a practical perspective, we consider the im-portance differences of the performance in different subareas and the coupled relationship between dif-ferent radar performance. To optimize the node placement scheme, we first establish evaluation metrics respectively for surveillance and localization performance. Then, we formulate a multi-objective geomet-ric optimization problem with complex coupled constraints. Considering that the final optimization prob-lem is difficult to tackle due to its high dimensionality, non-convexity, and especially the complex coupled constraint, we propose a novel self-constrained multi-objective particle swarm optimization (SC-MOPSO) algorithm for fast computation. The SC-MOPSO algorithm can be efficiently applied to the established op-timization problem with the complex coupled constraint satisfied. Moreover, the obtained Pareto-optimal solution set is more complete in comparison with the state-of-the-art algorithms. Finally, various numer-ical results show that the proposed method can effectively enhance both the system surveillance and localization performance while the complex coupled constraints are satisfied.(c) 2022 Elsevier B.V. All rights reserved.