Performance of chaotic communication in radio-over-fiber (ROF) transmission based on optoelectronic feedback semiconductor lasers is studied numerically. The chaotic carrier is generated by optoelectronic feedback semiconductor lasers, where chaotic communication is realized by synchronizing a receiver laser with a transmitter laser. Transmission quality of different message encoding schemes, including additive chaos modulation (ACM) and on-off shift keying (OOSK), are investigated and compared. In this study, the dispersion and nonlinearity effects in the fiber transmission module and the amplified spontaneous emission noise from the optical amplifiers are considered. In the wireless channel, effects of additive white Gaussian noise, multipath, and path loss are included. To quantitatively study the performance of this chaotic communication system in the ROF transmission, bit-error-rates (BER) of different transmission lengths, message bit-rates, and signal-to-noise ratios are studied. The optimal launched power and message strength that minimize the BER while assuring effective communication security are discussed. While the ACM scheme is shown to perform better in a fiber only configuration, the OOSK scheme shows better immunity to the random effects and waveform distortions presented in the wireless channel. (c) 2007 Optical Society of America.