Cross-lingual voice conversion based on F0 multi-scale modeling with VITS

This paper introduces a cross-lingual voice conversion technology that utilizes an F0 predictor for multi-scale modeling of the fundamental frequency (f0). Based on the VITS architecture, this method achieves high-quality voice generation through an end-to-end approach. In cross-lingual conversion, the voice often carries an unnatural foreign accent due to the involvement of different languages in the source and target voices. To address this issue, Whisper is introduced as a content extraction tool aimed at capturing the detailed content of the speech, including the specific accent of the source voice, which is crucial for efficient cross-lingual conversion. Furthermore, by employing an F0 predictor for multi-scale modeling of f0 to predict the fundamental frequency contour, this strategy helps to retain the accent characteristics of the source voice during conversion. This research, trained solely on the VCTK English dataset, effectively achieves cross-lingual conversion across various languages and significantly reduces the impact of foreign accents. To further explore the contribution of the F0 predictor in the proposed model, a series of ablation experiments were designed. Through objective and subjective evaluations, the effectiveness of the proposed method is demonstrated.