finetune-whisper-lora - GitLite

KO | EN

gitlite — search

#javascript #python #react #ai #llm #go #golang #machine-learning #docker #cli #awesome-list #pytorch #ios #open-source

finetune-whisper-lora

(★ 58)

Fine-Tune Whisper with Transformers and PEFT

File Explorer

Download Latest Version (.zip)

- - docker-compose.yml
  - Dockerfile.dev
  - requirements.txt
- - docker-compose.yml
  - Dockerfile
  - requirements.txt
- - .gitignore
  - infer.py
  - prepare_whisper_configs.py
  - README.md
  - whisper_template.json
- - .gitignore
  - benchmark.sh
  - bt_model_infer.py
  - onnx_model_infer.py
  - optimize_onnx_convert.py
  - optimized_model_infer.py
  - original_model_infer.py
  - quantize_onnx_convert.py
  - quantized_model_infer.py
  - README.md
  - vanilla_onnx_convert.py
- aishell_1.sh
- magicdata_mcrsc.sh
- mdcc.sh
- README.md
- thchs_30.sh
- docker-compose.yml
- Dockerfile.gpu
- requirements.txt
- docker-compose.yml
- Dockerfile.cpu
- requirements.txt
.gitignore
download_dataset.py
download_model.py
eval.py
eval_lora.py
finetune.py
finetune_lora.py
LICENSE
load_datasets.py
README.md
upload_model.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Finetune Whisper using LoRA for Cantonese and Mandarin

<p align="left">
🤗 <a href="https://huggingface.co/Oblivion208" target="_blank">HF Repo</a>  •🐱 <a href="https://github.com/fengredrum/finetune-whisper-lora" target="_blank">Github Repo</a> 
</p>

## Get Started

### 1. Setup Docker Environment

Switch to the docker folder and build Docker GPU image for training:

```bash
cd docker
docker compose build
```

Onece the building process complete, run the following command to start a Docker container and attach to it:

```bash
docker compose up -d
docker exec -it asr bash
```

### 2. Prepare Training Data

See detail in [dataset_scripts](dataset_scripts/README.md) folder.

### 3. Finetune Pretrained Model

```bash
# Finetuning
python finetune.py --model_id base --streaming True --train_batch_size 64 --gradient_accumulation_steps 2 --fp16 True
```

```bash
# LoRA Finetuning
python finetune_lora.py --model_id large-v2 --streaming True --train_batch_size 64 --gradient_accumulation_steps 2
```

### 4. Evaluate Performance

```bash
# Evaluation
python eval.py --model_name_or_path Oblivion208/whisper-tiny-cantonese --streaming True --batch_size 64
```

```bash
# LoRA Evaluation
python eval_lora.py --peft_model_id Oblivion208/whisper-large-v2-lora-mix --streaming True --batch_size 64
```

**Note:** Setting `--streaming` to `False` will cache acoustic features on local disk, which speeds up finetuning processes, but it increases the disk usage dramatically (almost three times of raw audio files size).

## Approximate Performance Evaluation

The following models are all trained and evaluated on a single RTX 3090 GPU via [Vast.ai](https://cloud.vast.ai/?ref_id=78038 "Vast.ai").

### Cantonese Test Results Comparison

#### MDCC

| Model name                      | Parameters | Finetune Steps | Time Spend | Training Loss | Validation Loss | CER % | Finetuned Model                                                                                                          |
| ------------------------------- | ---------- | -------------- | ---------- | ------------- | --------------- | ----- | ------------------------------------------------------------------------------------------------------------------------ |
| whisper-tiny-cantonese          | 39 M       | 3200           | 4h 34m     | 0.0485        | 0.771           | 11.10 | [Link](https://huggingface.co/Oblivion208/whisper-tiny-cantonese "Oblivion208/whisper-tiny-cantonese")                   |
| whisper-base-cantonese          | 74 M       | 7200           | 13h 32m    | 0.0186        | 0.477           | 7.66  | [Link](https://huggingface.co/Oblivion208/whisper-base-cantonese "Oblivion208/whisper-base-cantonese")                   |
| whisper-small-cantonese         | 244 M      | 3600           | 6h 38m     | 0.0266        | 0.137           | 6.16  | [Link](https://huggingface.co/Oblivion208/whisper-small-cantonese "Oblivion208/whisper-small-cantonese")                 |
| whisper-small-lora-cantonese    | 3.5 M      | 8000           | 21h 27m    | 0.0687        | 0.382           | 7.40  | [Link](https://huggingface.co/Oblivion208/whisper-small-lora-cantonese "Oblivion208/whisper-small-lora-cantonese")       |
| whisper-large-v2-lora-cantonese | 15 M       | 10000          | 33h 40m    | 0.0046        | 0.277           | 3.77  | [Link](https://huggingface.co/Oblivion208/whisper-large-v2-lora-cantonese "Oblivion208/whisper-large-v2-lora-cantonese") |

#### Common Voice Corpus 11.0

| Model name                      | Original CER % | w/o Finetune CER % | Jointly Finetune CER % |
| ------------------------------- | -------------- | ------------------ | ---------------------- |
| whisper-tiny-cantonese          | 124.03         | 66.85              | 35.87                  |
| whisper-base-cantonese          | 78.24          | 61.42              | 16.73                  |
| whisper-small-cantonese         | 52.83          | 31.23              | /                      |
| whisper-small-lora-cantonese    | 37.53          | 19.38              | 14.73                  |
| whisper-large-v2-lora-cantonese | 37.53          | 19.38              | 9.63                   |

## Requirements

- Transformers
- Accelerate
- Datasets
- PEFT
- bitsandbytes
- librosa

## References

1. https://github.com/openai/whisper
2. https://huggingface.co/blog/fine-tune-whisper
3. https://huggingface.co/docs/peft/task_guides/int8-asr
4. https://huggingface.co/alvanlii/whisper-largev2-cantonese-peft-lora