We describe an adaptive interferometer for linear detection of small out-of-plane vibrations of optically rough surfaces. The interferometer is based on the polarization self-modulation effect in a GaP crystal and it uses a laser diode at 807 nm as a light source. The achieved response time of 1.9 ms is the fastest among known adaptive interferometers in the near infrared.