Fractional-order operators on nonsmooth domains

The fractional Laplacian (-Delta)a$(-\Delta )<^>a$, a is an element of(0,1)$a\in (0,1)$, and its generalizations to variable-coefficient 2a$2a$-order pseudodifferential operators P$P$, are studied in Lq$L_q$-Sobolev spaces of Bessel-potential type Hqs$H<^>s_q$. For a bounded open set omega subset of Rn$\Omega \subset \mathbb {R}<^>n$, consider the homogeneous Dirichlet problem: Pu=f$Pu =f$ in omega$\Omega$, u=0$u=0$ in Rn set minus omega$ \mathbb {R}<^>n\setminus \Omega$. We find the regularity of solutions and determine the exact Dirichlet domain Da,s,q$D_{a,s,q}$ (the space of solutions u$u$ with f is an element of Hqs(omega over bar )$f\in H_q<^>s(\overline{\Omega })$) in cases where omega$\Omega$ has limited smoothness C1+tau$C<^>{1+\tau }$, for 2a<taus$-results obtained now when taua$. In detail, the spaces Da,s,q$D_{a,s,q}$ are identified as a$a$-transmission spaces Hqa(s+2a)(omega over bar )$H_q<^>{a(s+2a)}(\overline{\Omega })$, exhibiting estimates in terms of dist(x, partial differential omega)a$\operatorname{dist}(x,\partial \Omega )<^>a$ near the boundary.The result has required a new development of methods to handle nonsmooth coordinate changes for pseudodifferential operators, which have not been available before; this constitutes another main contribution of the paper.