CLASSIFICATION OF POSITIVE SOLUTIONS FOR AN ELLIPTIC SYSTEM WITH A HIGHER-ORDER FRACTIONAL LAPLACIAN

We discuss properties of solutions to the following elliptic PDE system in R-n: {(-Delta)(alpha/2)u = lambda(1)u(p1) + mu(1)v(p2) + beta(1)u(p3)v(p4), (-Delta)(alpha/2)v = lambda(2)u(q1) + mu(2)v(q2) + beta(2)u(q3)v(q4), where 0 < alpha < n, lambda(j), mu(j), beta(j) (j = 1, 2) are nonnegative constants and p(i) and q(i) (i = 1, 2, 3, 4) satisfy some suitable assumptions. It is shown that this PDE system is equivalent to the integral system {u(x) = integral(Rn) lambda(1)u(p1)(y) + mu(1)v(p2)(y) + beta(1)u(p3)(y)v(p4)(y)/vertical bar x - y vertical bar(n-alpha) dy, v(x) = integral(Rn) lambda(2)u(q1)(y) + mu(2)v(q2)(y) + beta(2)u(q3)(y)v(q4)(y)/vertical bar x - y vertical bar(n-alpha) dy in R-n. The radial symmetry, monotonicity and regularity of positive solutions are proved via the method of moving plane in integral forms and a regularity lifting lemma. For the special case with p(1) = p(2) = q(1) = q(2) = p(3) + p(4) = q(3) + q(4) = n + alpha/n - alpha, positive solutions of the integral system (or the PDE system) are classified. Furthermore, our symmetry results, together with some known results on nonexistence of positive solutions, imply that, under certain integrability conditions, the PDE system has no positive solution in the subcritical case.