Speech Synthesis Adaption Method Based on Phoneme-Level Speaker Embedding Under Small Data

In speech synthesis, the use of a small amount of user-recorded data for speaker adaptation has always been faced with a problem: how to synthesize highly similar speeches without excessively reducing the naturalness of the synthesized speeches. The existing utterance-level and frame-level speaker embedding methods face the problem of low similarity when synthesizing speeches of testing speaker, and the use of a small amount of user recorded data to fine-tune the pre-trained speech synthesis model can improve the similarity of synthesized audio, but it is often accompanied by a decrease in naturalness. To solve this problem, we propose a novel adaptation method for speech synthesis based on phoneme-level speaker embedding. In the training stage, the phoneme-level speaker embedding is extracted from the real feature fragments to control the training of the speech synthesis model. In the adaptation stage, we quickly adapt the speaker embedding predictor network, replacing the real audio in the inference stage to obtain phoneme-level speaker embedding. We use a small amount of real user-recorded data to conduct experiments, and compare the performance of common speaker embedding methods in different grains. Experiments show that compared with various speaker embedding methods, our method maintains no significant decrease in naturalness without updating the speech synthesis model, and achieves the best similarity; in the case of updating the speech synthesis model, our method achieves the best naturalness and similarity at the same time. The analysis found that the phoneme-level speaker embedding method provides a better initial point of model adaptation without increasing the adaptive training time, and effectively improves the quality of the synthesized speeches of the adaptive model. © 2022, Science Press. All right reserved.