by espnet
Open source · 735 downloads · 1 likes
This model combines FastSpeech2Conformer, a non-autoregressive text-to-speech system, with the HiFi-GAN vocoder to transform text into high-quality speech. It leverages the Conformer architecture, which merges the efficiency of FastSpeech2 with the benefits of convolutional and attention networks, to rapidly generate mel spectrograms. Using HiFi-GAN, these spectrograms are then converted into natural, fluid audio waveforms. Ideal for applications requiring fast and realistic speech synthesis—such as voice assistants, audiobooks, or assistive communication tools for the visually impaired—it stands out for its speed and superior sound quality compared to traditional models.
This model combines FastSpeech2Conformer and FastSpeech2ConformerHifiGan into one model for a simpler and more convenient usage.
FastSpeech2Conformer is a non-autoregressive text-to-speech (TTS) model that combines the strengths of FastSpeech2 and the conformer architecture to generate high-quality speech from text quickly and efficiently, and the HiFi-GAN vocoder is used to turn generated mel-spectrograms into speech waveforms.
You can run FastSpeech2Conformer locally with the 🤗 Transformers library.
pip install --upgrade pip
pip install --upgrade transformers g2p-en
from transformers import FastSpeech2ConformerTokenizer, FastSpeech2ConformerWithHifiGan
import soundfile as sf
tokenizer = FastSpeech2ConformerTokenizer.from_pretrained("espnet/fastspeech2_conformer")
inputs = tokenizer("Hello, my dog is cute.", return_tensors="pt")
input_ids = inputs["input_ids"]
model = FastSpeech2ConformerWithHifiGan.from_pretrained("espnet/fastspeech2_conformer_with_hifigan")
output_dict = model(input_ids, return_dict=True)
waveform = output_dict["waveform"]
sf.write("speech.wav", waveform.squeeze().detach().numpy(), samplerate=22050)