This paper examines solution methods for mathematical programs with complementarity constraints (MPCC) obtained from the time-discretization of optimal control problems (OCPs) subject to nonsmooth dynamical systems. The MPCC theory and stationarity concepts are reviewed and summarized. The focus is on relaxation-based methods for MPCCs, which solve a (finite) sequence of more regular nonlinear programs (NLP), where a regularization/homotopy parameter is driven to zero. Such methods perform reasonably well on currently available benchmarks. However, these results do not always generalize to MPCCs obtained from nonsmooth OCPs. To provide a more complete picture, this paper introduces a novel benchmark collection of such problems, which we call NOSBENCH. The problem set includes 603 different MPCCs and we split it into a few representative subsets to accelerate the testing. We compare different relaxation-based methods, NLP solvers, homotopy parameter update and relaxation parameter steering strategies. Moreover, we check whether the obtained stationary points allow first-order descent directions, which may be the case for some of the weaker MPCC stationarity concepts. In the best case, the Scholtes' relaxation (SIAM J. Optim. 11, 918-936, 2001) with IPOPT (Math. Program. 106, 25-57, 2006) as NLP solver manages to solve 73.8% of the problems. This highlights the need for further improvements in algorithms and software for MPCCs.