On a class of coupled Schrodinger systems with critical Sobolev exponent growth

Consider the following two critical nonlinear Schrodinger systems: {-Delta u + lambda(1)u = mu(1)u(2*-1) + alpha gamma/2*u(alpha-1)v(beta), x is an element of Omega, -Delta v + lambda(2)v = mu(2)v(2*-1) + beta gamma/2*u(alpha)v(beta-1), x is an element of Omega, (0.1) u > 0, v > 0 in Omega, u =v =0 on partial derivative Omega, {-Delta u = mu(1)vertical bar u vertical bar(2*-2)u + alpha gamma/2* f(alpha,beta)(u,v), x is an element of R-N, -Delta v = mu(2)vertical bar u vertical bar(2*-2)v + beta gamma/2* g(alpha,beta)(u,v), x is an element of R-N, (0.2) u, v in D-1,D-2 (R-N), where Omega subset of R-N is a smooth bounded domain, N >= 3, -lambda(Omega) < lambda(1), lambda(2) < 0, mu(1), mu(2) > 0, alpha, beta >= 1 with alpha + beta = 2*, gamma not equal 0, lambda(Omega) is the first eigenvalue of -Delta with the Dirichlet boundary condition and f(alpha,beta) (u, v) = {vertical bar u vertical bar(alpha-2)u vertical bar v vertical bar(beta), if alpha, beta > 1, vertical bar v vertical bar(2*-1), if alpha = 1, g(alpha, beta)(u,v) = {vertical bar u vertical bar(alpha)vertical bar v vertical bar(beta-2), if alpha, beta > 1, u vertical bar v vertical bar(2*-3)v, if alpha = 1, vertical bar u vertical bar(2*-1), if beta = 1. vertical bar u vertical bar(2*-3) uv, if beta = 1, For N = 3, lambda(1) = lambda(2), gamma > 0 small, we obtain the existence of positive least energy solution of (0.1) and ( 0.2). For N >= 5, gamma > 0, the existence of positive least energy solution of (0.2) is established. For N >= 5, gamma not equal 0, we prove that (0.1) possesses a positive least energy solution. The limit behavior of the positive least energy solutions when gamma -> -infinity and phase separation for (0.1) are also considered. Copyright (C) 2015 JohnWiley & Sons, Ltd.