QUASI-PERIODIC MOTION IN THE BILLIARD PROBLEM WITH A SOFTENED BOUNDARY

The motion of a classical particle in a convex plane region with softened boundary is considered. The Kolmogorov-Arnol'd-Moser theory is applied to detect ''whispering gallery'' trajectories, i.e., solutions staying near the boundary. It turns out that the large energy solutions starting near the boundary are quasiperiodic and stay there for all time filling out the invariant tori in the phase space, under some regularity conditions on the force repelling the particle from the boundary. The same technique is applied to the analysis of propagation of a skew light ray through the optical fiber with nonuniform core. It is shown that the majority of skew light rays starting near the boundary of the fiber stay there for all time under some restrictions on the refraction index.