NONLINEAR ELLIPTIC SYSTEMS INVOLVING THE FRACTIONAL LAPLACIAN IN THE UNIT BALL AND ON A HALF SPACE

In this paper, we study the following nonlinear elliptic system {(-Delta)(alpha/2)u(i) = f(i)(u), x epsilon Omega, i = 1, ..., m, u(i)(x) = 0, x epsilon Omega(c), i = 1, ..., m, where 0 < alpha < 2 and Omega is either the unit ball B-1(0) = {x epsilon R-n vertical bar parallel to x parallel to < 1} or the half space R-+(n) = {x = (x(1),...,x(n)) epsilon R-n vertical bar x(n) > 0}. Instead of investigating the pseudo differential system directly, we study an equivalent integral system, i.e., u(i)(x) = integral(B1(0)) G(1)(x,y)f(i)(u(y))dy, x epsilon B-1(0), i = 1, ..., m, and u(i)(x) = C(i)x(n)(alpha/2) + integral(R+n) G(infinity)(x,y)f(i)(u(y))dy, x epsilon R-+(n), i = 1, ..., m, where C-i are non-negative constants, G(1)(x,y) is Green's function for B-1(0) and G(infinity)(x,y) is Green function of R-+(n). We use the method of moving planes in integral forms to prove the radial symmetry and monotonicity of positive solutions in B-1(0) and non-existence of positive solutions in R-+(n). Moreover, we also study regularity of positive solutions in B-1(0).