In this paper, a cooperative hybrid visible light communication (VLC)-radio frequency (RF) relay system, which includes a transmitter (S) with a cluster of light emitting diodes (LED) lamps, a relay (R), and a destination (D), is taken into consideration. Moreover, decode-forward (DF) and amplify-forward (AF) schemes are considered at R to aid the data transmissions between S and D. Specially, R is randomly located in the coverage of the LED lamps, D is randomly distributed out of the coverage of the LED lamps. Then, VLC is employed for S-R link, and a traditional RF communication technology is adopted over the R-D link. Considering Nakagami-m fading over the R-D link, and the randomness of the locations of R and D, the statistical characteristics of the received signal-to-noise ratio at D under AF scheme, and the ones at R and D under the DF scheme are characterized, respectively. Subsequently, approximated analytical and asymptotic expressions for the outage probability of the considered system are derived, as well as the approximated expression for the average symbol error rate of M-ary quadrature amplitude modulation scheme under the DF scheme. Finally, simulation and numerical analysis are conducted to verify the correctness of our proposed models.