EDGE - GitLite

KO | EN

gitlite — search

#agent #ui #git #gemini #generative-ai #cloud-native #html #claude-code #ai-agent #redis

EDGE

(★ 575)

Official PyTorch Implementation of EDGE (CVPR 2023)

파일 탐색기

ZIP 다운로드

- - __init__.py
  - baseline_features.py
  - jukebox_features.py
- - crossmodal_test.txt
  - crossmodal_train.txt
  - ignore_list.txt
- __init__.py
- create_dataset.py
- download_dataset.sh
- filter_split_data.py
- slice.py
- __init__.py
- dance_dataset.py
- masks.py
- preprocess.py
- quaternion.py
- scaler.py
- __init__.py
- eval_pfc.py
- edge_cover.png
- adan.py
- diffusion.py
- model.py
- rotary_embedding_torch.py
- utils.py
- Convert.py
- FbxFormatConverter.exe
- FbxReadWriter.py
- SmplObject.py
- ybot.fbx
.gitignore
args.py
demo.ipynb
download_model.sh
EDGE.py
LICENSE
README.md
test.py
train.py
vis.py

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`accelerate`

설명 보기 ▼

Accelerate는 동일한 PyTorch 코드를 모든 분산 환경 구성에서 실행할 수 있게 해주는 라이브러리입니다.
더 많은 정보: <https://huggingface.co/docs/accelerate/index>.

accelerate env

실행환경 정보 출력:

accelerate config

대화형으로 구성 파일 생성:

accelerate estimate-memory {{name/model}}

다양한 데이터 타입을 사용하여 Hugging Face 모델을 실행하는 데 필요한 예상 GPU 메모리 비용을 출력:

🔍

`youtube-dl`

설명 보기 ▼

YouTube 및 다른 웹사이트에서 비디오를 다운로드.
관련 항목: `yt-dlp`, `ytfzf`, `you-get`.
더 많은 정보: <https://rg3.github.io/youtube-dl/>.

youtube-dl '{{https://www.youtube.com/watch?v=oHg5SJYRHA0}}'

비디오 또는 재생 목록 다운로드:

youtube-dl {{[-F|--list-formats]}} '{{https://www.youtube.com/watch?v=Mwa0_nE9H7A}}'

비디오 또는 재생 목록이 가능한 모든 형식 나열:

youtube-dl {{[-f|--format]}} "{{best[height<=480]}}" '{{https://www.youtube.com/watch?v=oHg5SJYRHA0}}'

특정 품질로 비디오 또는 재생 목록 다운로드:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

![EDGE cover image](https://raw.githubusercontent.com/Stanford-TML/EDGE/main/media/edge_cover.png)
## EDGE &mdash; Official PyTorch implementation
**EDGE: Editable Dance Generation From Music** (CVPR 2023)<br>
Jonathan Tseng, Rodrigo Castellon, C. Karen Liu<br>
https://arxiv.org/abs/2211.10658
*Abstract: Dance is an important human art form, but creating new dances can be difficult and time-consuming. In this work, we introduce Editable Dance GEneration (EDGE), a state-of-the-art method for editable dance generation that is capable of creating realistic, physically-plausible dances while remaining faithful to the input music. EDGE uses a transformer-based diffusion model paired with Jukebox, a strong music feature extractor, and confers powerful editing capabilities well-suited to dance, including joint-wise conditioning, and in-betweening. We introduce a new metric for physical plausibility, and evaluate dance quality generated by our method extensively through (1) multiple quantitative metrics on physical plausibility, beat alignment, and diversity benchmarks, and more importantly, (2) a large-scale user study, demonstrating a significant improvement over previous state-of-the-art methods.*
## Requirements
* We recommend Linux for performance and compatibility reasons. Windows will probably work, but is not officially supported.
* 64-bit Python 3.7+
* PyTorch 1.12.1
* At least 16 GB RAM per GPU
* 1&ndash;8 high-end NVIDIA GPUs with at least 16 GB of GPU memory, NVIDIA drivers, CUDA 11.6 toolkit.

The example build this repo was validated on:
* Debian 10
* 64-bit Python 3.7.12
* PyTorch 1.12.1
* 16 GB RAM
* 1 x NVIDIA T4, CUDA 11.6 toolkit

This repository additionally depends on the following libraries, which may require special installation procedures:
* [jukemirlib](https://github.com/rodrigo-castellon/jukemirlib)
* [pytorch3d](https://github.com/facebookresearch/pytorch3d)
* [accelerate](https://huggingface.co/docs/accelerate/v0.16.0/en/index)
	* Note: after installation, don't forget to run `accelerate config` . We use fp16.
* [wine](https://www.winehq.org) (Optional, for import to Blender only)
## Getting started
### Quickstart
* Download the saved model checkpoint from [Google Drive](https://drive.google.com/file/d/1BAR712cVEqB8GR37fcEihRV_xOC-fZrZ/view?usp=share_link) or by running `bash download_model.sh`.
* Run `demo.ipynb`, which demonstrates the basic interface of the model
### Load custom music
You can test the model on custom music by downloading them as `.wav` files into a directory, e.g. `custom_music/` and running
```.bash
python test.py --music_dir custom_music/
```
This process may take a while, since the script will extract all the Jukebox representations for the specified music in memory. The representations can also be saved and reused to improve speed with the `--cache_features` and `--use_cached_features` arguments. See `args.py` for more detail.
Note: make sure file names are regularized, e.g. `Britney Spears - Toxic (Official HD Video).wav` may cause unpredictable behavior due to the spaces and parentheses, but `toxic.wav` will behave as expected. See how the demo notebook achieves this using the `youtube-dl --output` flag.

### (Optional, retraining only) Dataset Download
Download and process the AIST++ dataset (wavs and motion only) using:
```.bash
cd data
bash download_dataset.sh
python create_dataset.py --extract-baseline --extract-jukebox
```
This will process the dataset to match the settings used in the paper. The data processing will take ~24 hrs and ~50 GB to precompute all the Jukebox features for the dataset.
### Train your own model
Once the AIST++ dataset is downloaded and processed, run the training script, e.g.
```.bash
accelerate launch train.py --batch_size 128  --epochs 2000 --feature_type jukebox --learning_rate 0.0002
```
to train the model with the settings from the paper. The training will log progress to `wandb` and intermittently produce sample outputs to visualize learning. Depending on the available GPUs, this can take ~6 - 24 hrs.
### Evaluate your model
Evaluate your model's outputs with the Physical Foot Contact (PFC) score proposed in the paper:
1. Generate ~1k samples, saving the joint positions with the `--save_motions` argument
2. Run the evaluation script
```.bash
python test.py --music_dir custom_music/ --save_motions
python eval/eval_pfc.py
```
## Blender 3D rendering
In order to render generated dances in 3D, we convert them into FBX files to be used in Blender. We provide a sample rig, `SMPL-to-FBX/ybot.fbx`.
After generating dances with the `--save-motions` flag enabled, move the relevant saved `.pkl` files to a folder, e.g. `smpl_samples`
Run
```.bash
python SMPL-to-FBX/Convert.py --input_dir SMPL-to-FBX/smpl_samples/ --output_dir SMPL-to-FBX/fbx_out
```
to convert motions into FBX files, which can be imported into Blender and retargeted onto different rigs, i.e. from [Mixamo](https://www.mixamo.com). A variety of retargeting tools are available, such as the [Rokoko plugin for Blender](https://www.rokoko.com/integrations/blender).

## Development
This is a research implementation and, in general, will not be regularly updated or maintained long after release.
## Citation
```
@article{tseng2022edge,
  title={EDGE: Editable Dance Generation From Music},
  author={Tseng, Jonathan and Castellon, Rodrigo and Liu, C Karen},
  journal={arXiv preprint arXiv:2211.10658},
  year={2022}
}
```
## Acknowledgements
We would like to thank [lucidrains](https://github.com/lucidrains) for the [Adan](https://github.com/lucidrains/Adan-pytorch) and [diffusion](https://github.com/lucidrains/denoising-diffusion-pytorch) repos, [softcat477](https://github.com/softcat477) for their [SMPL to FBX](https://github.com/softcat477/SMPL-to-FBX) library, and [BobbyAnguelov](https://github.com/BobbyAnguelov) for their [FBX Converter tool](https://github.com/BobbyAnguelov/FbxFormatConverter).