voicepuppet

KO | EN

gitlite — search

#javascript #hacktoberfest #android #rust #deep-learning #linux #macos #awesome #kubernetes #awesome-list #pytorch #mcp #openai #open-source

(★ 40)

Audio driven video synthesis

voicepuppet Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VoicePuppet #
 - This repository provided a common pipeline to generate speaking actor by voice input automatically.
 - For a better feeling, there's a [short video](https://youtu.be/h24MGPTTw5M) to demostrate it.

## The archecture of the network ##
 - Composed of 2 parts, one for predict 3D face coeffcients of each frame align to a certain stride window size of waveform, called BFMNet(basel face model network). The another for redraw the real face foreground using the rasterized face which produced by the rendered 3D face coeffcients of previous step, called PixReferNet.

<table>
  <tr>
    <th>
      <img src="https://github.com/taylorlu/voicepuppet/blob/master/res/1.png" >
    </th>
  </tr>
  <tr><th>
    BFMNet component
  </th></tr>
    <th>
      <img src="https://github.com/taylorlu/voicepuppet/blob/master/res/2.png" >
    </th>
  </tr>
  <tr><th>
    PixReferNet component
  </th></tr>
</table>

## Run the prediction pipeline ##
------------------------

1. Download the pretrained model and required models.</br>
    Baidu Disk: [[ckpt.zip](https://pan.baidu.com/s/1cVIVFhSsEA1MbgqL7H7mMw), code: a6pn], [[allmodels.zip](https://pan.baidu.com/s/11FKHjGjnPtD2c7Ttg-mXng), code: brfh]</br>
    or Google Drive: [[ckpt.zip](https://drive.google.com/file/d/1RgMSQUL2pzvwCWGgnkvwxHxHeEnZ7FlN/view?usp=sharing)], [[allmodels.zip](https://drive.google.com/file/d/1Z1Pm39sp977nED_HHZtvn5glRrmiThwB/view?usp=sharing)]</br>
    Extract the `ckpt.zip` to `ckpt_bfmnet` and `ckpt_pixrefer`, extract the `allmodels.zip` to current root dir
 2. `cd utils/cython` && `python3 setup.py install`
 3. Install ffmpeg tool if you want to merge the png sequence and audio file to video container like mp4.
 4. `python3 voicepuppet/pixrefer/infer_bfmvid.py --config_path config/params.yml sample/22.jpg sample/test.aac`

## Run the training pipeline ##
------------------------

#### Requirements ####

- tensorflow>=1.14.0
 - pytorch>=1.4.0, only for data preparation (face foreground segmentation and matting)
 - mxnet>=1.5.1, only for data preparation (face alignment)
 tips: you can use other models to do the same label marking instead, such as dlib

#### Data preparation ####

3. Face detection and alignment</br>
    `python3 datasets/make_data_from_GRID.py --gpu 0 --step 3 srcdir todir ./allmodels`

4. 3D face reconstruction</br>
    `python3 datasets/make_data_from_GRID.py --gpu 0 --step 4 todir ./allmodels`

5. It will take several hours to finish the above steps, subsequently, you'll find there's `*.jpg, landmark.txt, audio.wav, bfmcoeff.txt` in each output subfolder. The above labels(`audio.wav`, `bfmcoeff.txt`) are used for BFMNet training, the others are only temp files.
```
|- todir/
|    |- s10/
|        |- bbab8n/
|            |- landmark.txt
|            |- audio.wav
|            |- bfmcoeff.txt
|            |- 0.jpg
|            |- 1.jpg
|            |- ...
|        |- bbab9s/
|            |- ...
|    |- s8/
|        |- bbae5n/
|            |- landmark.txt
|            |- audio.wav
|            |- bfmcoeff.txt
|            |- 0.jpg
|            |- 1.jpg
|            |- ...
|        |- bbae6s/
|            |- ...
```
 6. Face(human foreground) segmentation and matting for PixelReferNet training. Before invoke the python shell, you should make sure the width and height of the video was in the same size(1:1). In general, 3-5 minutes video was enough for training the PixelReferNet network, the trained model will only take effect on this specified person too.</br>
    `python3 datasets/make_data_from_GRID.py --gpu 0 --step 6 src_dir to_dvp_dir ./allmodels`</br>
the `src_dir` has the same folder structure as [tip1 in Data preparation], when finish the above step, you will find `*.jpg` in subfolders, like this
<div align="center">
<img src="https://github.com/taylorlu/voicepuppet/blob/master/sample/22.jpg">
</div>

#### Train BFMNet ####

1. Prepare train and eval txt, check the `root_path` parameter in `config/params.yml` is the output folder of [tip1 in Data preparation]</br>
    `python3 datasets/makelist_bfm.py --config_path config/params.yml`
2. train the model</br>
    `python3 voicepuppet/bfmnet/train_bfmnet.py --config_path config/params.yml`
3. Watch the evalalute images every 1000 step in `log/eval_bfmnet`, the upper was the target sequence, and the under was the evaluated sequence.
<div align="center">
<img src="https://github.com/taylorlu/voicepuppet/blob/master/res/3.jpg">
</div>

#### Train PixReferNet ####

1. Prepare train and eval txt, check the `root_path` parameter in `config/params.yml` is the output folder of [tip6 in Data preparation]</br>
    `python3 datasets/makelist_pixrefer.py --config_path config/params.yml`
2. train the model</br>
    `python3 voicepuppet/pixrefer/train_pixrefer.py --config_path config/params.yml`
3. Use tensorboard to watch the training process</br>
    `tensorboard --logdir=log/summary_pixrefer`

## Acknowledgement ##
1. The face alignment model was refer to [Deepinx's work](https://github.com/deepinx/deep-face-alignment), it's more stable than Dlib.
2. 3D face reconstruction model was refer to [microsoft's work](https://github.com/microsoft/Deep3DFaceReconstruction)
3. Image segmentation model was refer to [gasparian's work](https://github.com/gasparian/PicsArtHack-binary-segmentation)
4. Image matting model was refer to [foamliu's work](https://github.com/foamliu/Deep-Image-Matting)

voicepuppet

voicepuppet Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

voicepuppet

voicepuppet Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README