Multiple positive solutions for nonlinear coupled fractional Laplacian system with critical exponent

In this paper, we study the following critical system with fractional Laplacian: {(-Delta)(s)u + lambda(1)u = mu(1)vertical bar u vertical bar(2)*(-2)u + alpha gamma/2*vertical bar u vertical bar(alpha-2)u vertical bar v vertical bar(beta) in Omega, (-Delta)(s)v + lambda(2)v = mu(2)vertical bar v vertical bar(2)*(-2)v + beta gamma/2*vertical bar u vertical bar(alpha)vertical bar v vertical bar(beta-2)v in Omega, u = v = 0 in R-N\Omega, where (-Delta)(s) is the fractional Laplacian, 0 < s < 1, mu(1), mu(2) > 0, 2* = 2N/N-2s is a fractional critical Sobolev exponent, N > 2s, 1 < alpha, beta < 2, alpha + beta = 2*, Omega is an open bounded set of R-N with Lipschitz boundary and lambda(1), lambda(2) > -lambda(1,s)(Omega), lambda(1,s)(Omega) is the first eigenvalue of the non-local operator (-Delta)(s) with homogeneous Dirichlet boundary datum. By using the Nehari manifold, we prove the existence of a positive ground state solution of the system for all gamma > 0. Via a perturbation argument and using the topological degree and a pseudo-gradient vector field, we show that this system has a positive higher energy solution. Then the asymptotic behaviors of the positive ground state solutions are analyzed when gamma -> 0.