Automatic-Prosody-Annotator-with-SSWP-CLAP - GitLite

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #machine-learning #rust #deep-learning #java #cli #kubernetes #awesome-list #vue #pytorch #security

Automatic-Prosody-Annotator-with-SSWP-CLAP

(★ 50)

An automatic prosodic boundary annotation tool for Text-to-Speech Synthesis (TTS).

파일 탐색기

최종 버전 다운로드 (.zip)

- prosodic_boundaries_labels_nopunc.json
- - dsp.py
  - TTS_CLAP_16k.yaml
- 01_wav2mel.py
- 01_inference_prosody_annotation.sh
- .DS_Store
- ablation.png
- model_v2.3_trimmed.png
- mos.png
- objective.png
- pref.png
- finetuned_prosody_annotator.pt
- - __init__.py
  - bpe_simple_vocab_16e6.txt.gz
  - factory.py
  - feature_fusion.py
  - htsat.py
  - linear_probe_words2sent.py
  - model.py
  - pann_model.py
  - tokenizer.py
  - utils.py
  - version.py
- - data_words2sent.py
  - logger.py
  - lp_evaluate_words2sent.py
  - lp_main_words2sent.py
  - params.py
- __init__.py
.gitattributes
LICENSE
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/jzmzhong/Automatic-Prosody-Annotator-with-SSWP-CLAP

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd Automatic-Prosody-Annotator-with-SSWP-CLAP

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Multi-modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

This is the official implementation of the following paper ([Arxiv](https://arxiv.org/abs/2309.05423)):

> **Multi-Modal Automatic Prosody Annotation with Contrastive Pretaining of SSWP** \
> Jinzuomu Zhong, Yang Li, Hui Huang, Korin Richmond, Jie Liu, Zhiba Su, Jing Guo, Benlai Tang, Fengjie Zhu

> **Abstract**: *In expressive and controllable Text-to-Speech (TTS), explicit prosodic features significantly improve the naturalness and controllability of synthesised speech. However, manual prosody annotation is labor-intensive and inconsistent. To address this issue, a two-stage automatic annotation pipeline is novelly proposed in this paper. In the first stage, we use contrastive pretraining of Speech-Silence and Word-Punctuation (SSWP) pairs to enhance prosodic information in latent representations. In the second stage, we build a multi-modal prosody annotator, comprising pretrained encoders, a text-speech fusing scheme, and a sequence classifier. Experiments on English prosodic boundaries demonstrate that our method achieves state-of-the-art (SOTA) performance with 0.72 and 0.93 f1 score for Prosodic Word and Prosodic Phrase boundary respectively, while bearing remarkable robustness to data scarcity. *

# Model Architecture

<h3>The Architecture of Our Proposed Work</h3>
<figure>
<p style="text-align:center"><img src="./figs/model_v2.3_trimmed.png" alt="AVSE" height=500px/></p>
</figure>

# Results & Demos

<h3>Objective Evaluation</h3>
The results of our proposed work, compared with previous benchmarks, are shown below.
<figure>
<p style="text-align:center"><img src="./figs/objective.png" alt="AVSE" height=120px/></p>
</figure>

Audio Samples Demo are avaialble at: [Demo](https://jzmzhong.github.io/Automatic-Prosody-Annotator-With-SSWP-CLAP/)

# Quickstart

## Environment Installation

```bash
conda create -n clap python=3.10
conda activate clap
# you can also install pytorch by following the official instruction (https://pytorch.org/get-started/locally/)
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0+cu113 -f https://download.pytorch.org/whl/torch_stable.html
pip install -r requirements.txt
```

## Multi-modal Prosody Annotation

### Data Process

The model requires the input data to be aligned using Kaldi. After that, it needs to be converted to the format of the sample data.

Text Features are stroed in *.json; Audio features are stored in *.mel.npy.

```shell
./sample_data/prosody_annotation
├── wordboundarylevel_ling
│   └── internal-spk1-test
└── wordboundarylevel_mel
    └── internal-spk1-test
```

To convert audio to mel, you can refer to the following command:

```base
cd data_process
python 01_wav2mel.py
```

### Sample Inference Script

```base
bash ./example/01_inference_prosody_annotation.sh
```
### Released Multi-modal Prosody Annotator

```
./released_model/finetuned_prosody_annotator.pt
```

# References

We are grateful for the following open-source contributions. Most of our codes are based on [LAION-AI's CLAP](https://github.com/LAION-AI/CLAP), with the Conformer part from [ESPNET](https://github.com/espnet/espnet).

- [LAION-AI's CLAP](https://github.com/LAION-AI/CLAP)
- [ESPNET](https://github.com/espnet/espnet)
- [CLAPSpeech](https://clapspeech.github.io/)
- [Automatic Prosody Annotation](https://github.com/Daisyqk/Automatic-Prosody-Annotation)