Fine-tuning and inference the Flowtron model which is an auto-regressive flow-based generative network for text to speech synthesis with control over speech variation and style transfer.
Download the Resources:
Download this resources manually from top right side of this page or by using wget command:
wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/flowtron/versions/2/zip -O flowtron_2.zip
Initialize Submodules:
The second step is initializing the submodule using these commands:
cd flowtron
git submodule update --init
cd tacotron2
git submodule update --init
Download the Dataset:
We need to Download a dataset for training, fine tuning and inference. You can find a dataset from this address :
"https://keithito.com/LJ-Speech-Dataset/ "
Edit the train and validation metadata file in the filelists based on the path to the dataset. Besides, you need to change the name of train and validation metadata text file in config.json file in data-config section.
Download Model Checkpoints:
Find flowtron checkpoints in ngc/models/Flowtron at this address: " https://ngc.nvidia.com/catalog/models/nvidia:adlr:flowtron ". Download them , make a folder inside the Flowtron folder and call it models then put these checkpoints in this folder. Find the WaveGlow checkpoints in ngc/models/WaveGlow at this address: " https://ngc.nvidia.com/catalog/models/nvidia:adlr:waveglow ". Download and put it in the models folder inside the Flowtron folder.
Build the Container:
The next step is building a container using the Dockerfile inside flowtron directory. This will make sure that we have access to all necessary resources for running the model in this container environment. Besides, it installs all models requirements.
Docker build -t flowtron .
Run the Container:
Now we have a container with a dataset and all requirements inside it. Run the container using following command:
docker run -it --runtime=nvidia -e NVIDIA_DRIVER_CAPABILITIES=compute,utility -e NVIDIA_VISIBLE_DEVICES=all flowtron
Fine-tuning:
You can fine tune the model using the downloaded dataset by this command:
python train.py -c config.json -p train_config.finetune_layers=["speaker_embedding.weight"] train_config.checkpoint_path="/workspace/flowtron/models/flowtron_libritts2p3k.pt"
After running the finetune command you can see the saved checkpoints of the mode in a folder called outdir.
Inference the Model: Use this command to inference the model using weights of fine tuned model:
python inference.py -c config.json -f /workspace/flowtron/models/model_path
-w /workspace/flowtron/models/waveglow_256channels_universal_v5.pt -t "It is well known that deep generative models have a rich latent space!" -i 0 -n 1000
After running inference you can see the result audio in the results folder. You can change the text in this command to any text that you want to creat.
Inference Examples: Following is some examples of inference the flowtron model using different flowtron checkpoints from https://ngc.nvidia.com/catalog/models/nvidia:adlr:flowtron .
- LJSpeech
python inference.py -c config_Flowtron-LJSpeech.json -f models/flowtron_ljs.pt -w models/waveglow_256channels_universal_v5.pt -t "Speech test" -i 0 -o results/Flowtron-LJSpeech
- Flowtron-LibriTTS
python inference.py -c config_Flowtron-LibriTTS.json -f models/flowtron_libritts.pt -w models/waveglow_256channels_universal_v5.pt -t "Speech test" -i 1088 -o results/Flowtron-LibriTTS
3.Flowtron-LibriTTS-2p3k
python inference.py -c config_Flowtron-LibriTTS-2p3k.json -f models/flowtron_ljs.pt -w models/waveglow_256channels_universal_v5.pt -t "Speech test" -i 1088 -o results/Flowtron-LibriTTS-2p3k
4.Flowtron-LibriTTS-2p3k_Fine-Tuned
python inference.py -c config_Flowtron-LibriTTS-2p3k_Fine-Tuned.json -f models/model_121000 -w models/waveglow_256channels_universal_v5.pt -t "Speech test" -i 1088 -o results/Flowtron-LibriTTS-2p3k_Fine-Tuned