In the paper, the shallow water equations are applied to describe the propagation of long waves in the coastal area of an ocean. For a correct formulation of the problem, the equations are closed by boundary conditions involving a function on the open water boundary. In general case this function is unknown. The determination of this function is reduced to the solution of the inverse problem on restoring it with auxiliary data on elevation of the sea surface along some part of the boundary. The solving this (ill-posed) inverse problem is performed by optimal controlmethods using adjoint operators. To improve the conditioning of the problem, three types of regularization functionals are considered which correspond to higher, deficient, and threshold smoothness of the data involved. The results of their application are illustrated by a numerical example. © Springer International Publishing Switzerland 2015.