PhysPT - GitLite

KO | EN

gitlite — search

#ai #typescript #go #machine-learning #rust #deep-learning #docker #windows #awesome #kubernetes #vue #mcp #openai #open-source

PhysPT

(★ 55)

Repositoray provides the PhysPT demo code for estimating human dynamics from a monocular video, CVPR2024

파일 탐색기

최종 버전 다운로드 (.zip)

- config.cpython-37.pyc
- config.cpython-38.pyc
- constants.cpython-37.pyc
- constants.cpython-38.pyc
- - dataloader_cliff.cpython-38.pyc
  - utils.cpython-37.pyc
  - utils.cpython-38.pyc
- - .DS_Store
- - .DS_Store
- .DS_Store
- dataloader_cliff.py
- utils.py
- - 000001.jpg
  - 000002.jpg
  - 000003.jpg
  - 000004.jpg
  - 000005.jpg
  - 000006.jpg
  - 000007.jpg
  - 000008.jpg
  - 000009.jpg
  - 000010.jpg
  - 000011.jpg
  - 000012.jpg
  - 000013.jpg
  - 000014.jpg
  - 000015.jpg
  - 000016.jpg
  - 000017.jpg
  - 000018.jpg
  - 000019.jpg
  - 000020.jpg
  - 000021.jpg
  - 000022.jpg
  - 000023.jpg
  - 000024.jpg
  - 000025.jpg
  - 000026.jpg
  - 000027.jpg
  - 000028.jpg
  - 000029.jpg
  - 000030.jpg
  - 000031.jpg
  - 000032.jpg
  - 000033.jpg
- jumpingjacks.json
- jumpingjacks_CLIFF.json
- jumpingjacks_output.npz
- - __init__.cpython-37.pyc
  - __init__.cpython-38.pyc
  - GlobalTrajPredictor.cpython-37.pyc
  - GlobalTrajPredictor.cpython-38.pyc
  - PhysPT.cpython-37.pyc
  - PhysPT.cpython-38.pyc
  - smpl_phys.cpython-37.pyc
  - smpl_phys.cpython-38.pyc
- - resnet.py
- - .DS_Store
  - cliff.py
  - smpl_mean_params.npz
- .DS_Store
- __init__.py
- GlobalTrajPredictor.py
- PhysPT.py
- smpl_phys.py
.DS_Store
config.py
constants.py
LICENSE
method.png
README.md
requirements.txt
video_inference.py
video_preprocessing.py
vis_motion_force_withvideo.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# **<ins>PhysPT</ins>: <ins>Phys</ins>ics-aware <ins>P</ins>retrained <ins>T</ins>ransformer for Estimating Human Dynamics from Monocular Videos** <br />
  [Yufei Zhang](https://zhangy76.github.io/), Jeffrey O. Kephart, Zijun Cui, Qiang Ji <br /> 
  CVPR2024, [arXiv](https://arxiv.org/abs/2404.04430) <br />
![](https://raw.githubusercontent.com/zhangy76/PhysPT/main/method.png)

This repositoray provides the PhysPT demo code for estimating human dynamics from a monocular video.

## Environment Setup
```bash
conda create -n physpt python=3.7
conda activate physpt
pip install -r requirements.txt
```

## Model and Data Download
Please download the required data and trained model [assets](https://www.dropbox.com/scl/fo/x9yor045ztrcv6rav4pz2/AMDO2krghJF_spnxNc5d9nE?rlkey=epj0vkwoafyp48rz7lp5im1o0&dl=0)  (due to license restrictions, please download SMPL and SMPL-H model from the [official website](https://smpl-x.is.tue.mpg.de/) and store them as ./assets/data/smpl/neutral.pkl; ./assets/data/smplh/female.npz; ./assets/data/smplh/male.npz; ./assets/data/smplh/neutral.npz) and directly overwrite the ./assets folder in the current directory. (please also download the checkpoint [CLIFF](https://www.dropbox.com/scl/fi/1zyefhokc2seull2un8bt/hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pt?rlkey=0hidvwxdi7g7770it04ncalyh&dl=0) and put it under ./models/cliff_hr48 used for generating kinematics-based motion estimates)

## Evaluation on a Video
#### 1. Estimate per-frame 3D Body Pose and Shape (CLIFF)
```bash
python video_preprocessing.py --vid_path './demo/jumpingjacks'
```
#### 2. Generate Improved 3D Motion Estimates and Infer Forces
```bash
python video_inference.py --vid_processed_path './demo/jumpingjacks_CLIFF.json'
```
#### 3. Visualize the Motion and Forces 
```bash
python vis_motion_force_withvideo.py --vid_output_path './demo/jumpingjacks_output.npz'
```

## Citation
If you find our work useful, please consider citing the paper:
```bibtex
@InProceedings{Zhang_2024_CVPR,
    author    = {Zhang, Yufei and Kephart, Jeffrey O. and Cui, Zijun and Ji, Qiang},
    title     = {PhysPT: Physics-aware Pretrained Transformer for Estimating Human Dynamics from Monocular Videos},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {2305-2317}
}
```

If you have questions or encouter any issues when running the code, feel free to open an issue or directly contact me via: zhangy76@rpi.edu.

## References
The SMPL model data is downloaded from [SMPL-X model](https://github.com/vchoutas/smplx). The adaptation of the CLIFF model is based on [CLIFF](https://github.com/haofanwang/CLIFF). We thank them for generously sharing their outstanding work.