We propose a scheme to realize frequency conversion of photons, using a hybrid system with a qubit coupled to both a resonator cavity and an optomechanical cavity. In this system, a high-energy photon can be converted into a low-energy photon plus a phonon and vice versa; the qubit works as an intermediate medium with its states unchanged. Moreover, the process can be exquisitely tuned by changing the frequency of the qubit or the resonator mode, providing additional control for the conversion. We further analyze the system dynamics and demonstrate two Rabi oscillation behaviors with dissipations. With the state-of-the-art techniques in cavity quantum electrodynamics and optomechanical systems, our proposal would be implemented and an efficient and controllable quantum light source can be achieved.