VAENAR-TTS - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #ai #llm #golang #android #linux #docker #macos #awesome #kubernetes #vue #pytorch #security

VAENAR-TTS

(★ 74)

PyTorch Implementation of VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis.

파일 탐색기

최종 버전 다운로드 (.zip)

- __init__.py
- audio.py
- - model.yaml
  - preprocess.yaml
  - train.yaml
- __init__.py
- config.json
- generator_LJSpeech.pth.tar.zip
- generator_universal.pth.tar.zip
- LICENSE
- models.py
- LJSpeech_val_dec_attn_0_LJ029-0157.gif
- LJSpeech_val_LJ029-0157.gif
- model.png
- tensorboard_audio.png
- tensorboard_loss.png
- tensorboard_spec.png
- librispeech-lexicon.txt
- pinyin-lexicon-r.txt
- __init__.py
- attention.py
- decoder.py
- encoder.py
- flow.py
- glow.py
- length_predictor.py
- optimizer.py
- posterior.py
- prior.py
- transform.py
- utils.py
- VAENAR.py
- - speakers.json
  - train.txt
  - val.txt
- ljspeech.py
- preprocessor.py
- __init__.py
- cleaners.py
- cmudict.py
- numbers.py
- pinyin.py
- symbols.py
- model.py
- tools.py
.gitignore
dataset.py
evaluate.py
LICENSE
prepare_align.py
preprocess.py
README.md
requirements.txt
synthesize.py
train.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/keonlee9420/VAENAR-TTS

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd VAENAR-TTS

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip3 install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VAENAR-TTS - PyTorch Implementation

PyTorch Implementation of [VAENAR-TTS: Variational Auto-Encoder based Non-AutoRegressive Text-to-Speech Synthesis](https://arxiv.org/abs/2107.03298).

<p align="center">
    <img src="img/model.png" width="80%">
</p>

The validation logs up to 70K of synthesized mel and alignment are shown below (LJSpeech_val_dec_attn_0_LJ029-0157 and LJSpeech_val_step_LJ029-0157 from top to bottom).

<p align="center">
    <img src="./img/LJSpeech_val_dec_attn_0_LJ029-0157.gif" width="80%">
</p>

<p align="center">
    <img src="./img/LJSpeech_val_LJ029-0157.gif" width="80%">
</p>

# Quickstart

## Dependencies
You can install the Python dependencies with
```
pip3 install -r requirements.txt
```

## Inference

You have to download the [pretrained models](https://drive.google.com/drive/folders/1-zgh0ltDHfjYT0i5xsNCjg-MsDDrhga7?usp=sharing) and put them in ``output/ckpt/LJSpeech/``.

For English single-speaker TTS, run
```
python3 synthesize.py --text "YOUR_DESIRED_TEXT" --restore_step RESTORE_STEP --mode single -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
```
The generated utterances will be put in ``output/result/``.

## Batch Inference
Batch inference is also supported, try

```
python3 synthesize.py --source preprocessed_data/LJSpeech/val.txt --restore_step RESTORE_STEP --mode batch -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
```
to synthesize all utterances in ``preprocessed_data/LJSpeech/val.txt``

# Training

## Datasets

The supported datasets are

- [LJSpeech](https://keithito.com/LJ-Speech-Dataset/): a single-speaker English dataset consists of 13100 short audio clips of a female speaker reading passages from 7 non-fiction books, approximately 24 hours in total.

## Preprocessing
 
First, run 
```
python3 prepare_align.py config/LJSpeech/preprocess.yaml
```
for some preparations. And then run the preprocessing script.
```
python3 preprocess.py config/LJSpeech/preprocess.yaml
```

## Training

Train your model with
```
python3 train.py -p config/LJSpeech/preprocess.yaml -m config/LJSpeech/model.yaml -t config/LJSpeech/train.yaml
```

# TensorBoard

Use
```
tensorboard --logdir output/log/LJSpeech
```

to serve TensorBoard on your localhost.
The loss curves, synthesized mel-spectrograms, and audios are shown.

![](https://raw.githubusercontent.com/keonlee9420/VAENAR-TTS/main/img/tensorboard_loss.png)
![](https://raw.githubusercontent.com/keonlee9420/VAENAR-TTS/main/img/tensorboard_spec.png)
![](https://raw.githubusercontent.com/keonlee9420/VAENAR-TTS/main/img/tensorboard_audio.png)

# Implementation Issues

- Removed arguments, methods during converting Tensorflow to PyTorch: name, kwargs, training, get_config()
- Specify `in_features` in **LinearNorm** which is corresponding to `tf.keras.layers.Dense`. Also, `in_channels` is explicitly specified in **Conv1D**.
- get_mask_from_lengths() function returns logical not of that of FastSpeech2.
- In this implementation, the `griffin_lim` algorithms is used to convert a mel-spectrogram to a waveform. You can use **HiFi-GAN** as a vocoder by setting config, but you need to [train it from scratch](https://github.com/jik876/hifi-gan) (you cannot use the provided pre-trained **HiFi-GAN** model).

# Citation

```
@misc{lee2021vaenar-tts,
  author = {Lee, Keon},
  title = {VAENAR-TTS},
  year = {2021},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/keonlee9420/VAENAR-TTS}}
}
```

# References
- [thuhcsi's VAENAR-TTS](https://github.com/thuhcsi/VAENAR-TTS)