In numerical simulations of black hole binaries, Pretorius and Khurana [Classical Quantum Gravity 24, S83 (2007)] have observed critical behavior at the threshold between scattering and immediate merger. The number of orbits scales as n similar or equal to -gamma ln vertical bar p - p(*)vertical bar along any one-parameter family of initial data such that the threshold is at p = p(*). Hence, they conjecture that in ultrarelativistic collisions almost all the kinetic energy can be converted into gravitational waves if the impact parameter is fine-tuned to the threshold. As a toy model for the binary, they consider the geodesic motion of a test particle in a Kerr black hole spacetime, where the unstable circular geodesics play the role of critical solutions, and calculate the critical exponent gamma. Here, we incorporate radiation reaction into this model using the self-force approximation. The critical solution now evolves adiabatically along a sequence of unstable circular geodesic orbits under the effect of the self-force. We confirm that almost all the initial energy and angular momentum are radiated on the critical solution. Our calculation suggests that, even for infinite initial energy, this happens over a finite number of orbits given by n(infinity) similar or equal to 0.41/eta, where eta is the (small) mass ratio. We derive expressions for the time spent on the critical solution, number of orbits and radiated energy as functions of the initial energy and impact parameter.