LaMo

KO | EN

gitlite — search

#pdf #nodejs #android #cpp #json #cli #containers #animation #windows #devops

(★ 9)

Self-Supervised Latent Motion Priors for Physical Realism in Video Generation

파일 탐색기

ZIP 다운로드

- - 0-1.gif
  - 0-2.gif
  - 1-1.gif
  - 1-2.gif
  - 2-1.gif
  - 2-2.gif
  - 3-1.gif
  - 3-2.gif
  - 4-1.gif
  - 4-2.gif
  - 5-1.gif
  - 5-2.gif
  - 6-1.gif
  - 6-2.gif
  - 7-1.gif
  - 7-2.gif
- - 0-1.mp4
  - 0-2.mp4
  - 1-1.mp4
  - 1-2.mp4
  - 2-1.mp4
  - 2-2.mp4
  - 3-1.mp4
  - 3-2.mp4
  - 4-1.mp4
  - 4-2.mp4
  - 5-1.mp4
  - 5-2.mp4
  - 6-1.mp4
  - 6-2.mp4
  - 7-1.mp4
  - 7-2.mp4
- - interp.png
  - interp_supp.png
  - method_v2.png
  - motion_drift_loss_icon.png
  - teaser-v1.png
- example.json
- getting_started.md
- - model_config_eval.yaml
  - model_config_vbench_eval.yaml
  - model_config_videophy2_eval.yaml
  - requirements_vbench.txt
- - _run_vbench_eval_entry.py
  - _run_videophy2_eval_entry.py
  - _run_videophy_eval_entry.py
  - download_vbench_pretrained.sh
  - download_videocon_physics.sh
  - download_videophy2_auto.sh
  - normalize_lamo_predictor_keys.py
  - run_cluster_eval_local.sh
  - run_vbench_eval.sh
  - run_videophy2_eval.sh
  - run_videophy_eval.sh
- - dimension_to_folder.json
  - evaluate.py
  - evaluate.sh
  - README.md
  - requirements.txt
  - setup.py
  - static_filter.py
- - .gitignore
  - LICENSE
  - README.md
  - requirements.txt
- - entailment_eval_unified.py
  - inference.py
  - README.md
  - template.py
  - utils.py
- __init__.py
- cluster_eval.py
- cluster_vbench_eval.py
- cluster_videophy2_eval.py
- - __init__.py
  - diffusion.py
  - image.py
  - text.py
  - video.py
- - __init__.py
  - modeling_utils.py
  - utils.py
- - __init__.py
  - accelerate.py
  - base.py
  - deepspeed.py
  - ptd.py
  - utils.py
- - __init__.py
  - utils.py
- - __init__.py
  - base.py
  - clip.py
  - llama.py
  - t5.py
  - text.py
- - __init__.py
  - config_utils.py
- - __init__.py
  - _common.py
  - activation_checkpoint.py
  - data.py
  - diffusion.py
  - file.py
  - hub.py
  - import_utils.py
  - memory.py
  - model.py
  - state_checkpoint.py
  - torch.py
- __init__.py
- args.py
- config.py
- constants.py
- data.py
- logging.py
- optimizer.py
- state.py
- trackers.py
- typing.py
- __init__.py
- _generation_common.py
- _interpret_common.py
- generate_cogvideox.py
- generate_lamo.py
- interpret_lamo.py
- vbench_cluster_generate.py
- videophy_cluster_generate.py
- - compiled_1.yaml
  - deepspeed.yaml
  - uncompiled_1.yaml
  - uncompiled_2.yaml
  - uncompiled_4.yaml
  - uncompiled_8.yaml
- - model_config_train_eval.yaml
- - run_cluster_train_local_lamo.sh
- __init__.py
- cluster_train_eval.py
- train.py
- dist.py
- model_path_resolver.py
- wandb_utils.py
.gitignore
LICENSE
NOTICE
pyproject.toml
README.md
requirements.txt
setup.py

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

LaMo improves physical realism in video generation by learning motion priors from unlabeled videos.

<a href="https://scholar.google.com/citations?user=UlDxGP0AAAAJ">Bo Jiang</a>1,*,
 <a href="https://scholar.google.com/citations?user=2oYPT9IAAAAJ">Depu Meng</a>1,
 <a href="https://scholar.google.com/citations?user=lf0bLigAAAAJ">Yihan Hu</a>1,
 <a href="https://scholar.google.com/citations?user=SdX6DaEAAAAJ">Yichen Xie</a>1,2,*,
 <a href="https://scholar.google.com/citations?user=I6_dXvEAAAAJ">Tianshuo Xu</a>1,*,
 <a href="https://scholar.google.com/citations?user=xVN3UxYAAAAJ">Wei Zhan</a>1,2

1Applied Intuition &nbsp;&nbsp; 2University of California, Berkeley
 
 *Work done during internship at Applied Intuition.

## News

- `[2026-06]` Training, inference, and evaluation code released.
- `[2026-05]` LaMo paper is available on arXiv.

## Highlights & Introduction

This repo contains the implementation of **LaMo**, including training, inference, evaluation, and interpretability scripts.

LaMo is designed for improving motion and physical consistency in text-to-video diffusion models:

- **Self-supervised motion prior** learned from ordinary unlabeled videos.
- **No external simulators, teacher models, or physics annotations** required.
- **Plug-and-play design** for existing video diffusion backbones.
- **Training-time and inference-time components** that can be used together or independently.
- **Improved physical commonsense** on VideoPhy and VideoPhy2.
- **Preserved general video quality** on VBench.

## Qualitative Results

The examples below compare CogVideoX-5B and LaMo-5B on physical motion prompts.
Click a preview to open the MP4.

| Prompt | CogVideoX-5B | LaMo-5B |
|---|---|---|
| Milk spilling into coffee creates waves. | [![CogVideoX milk coffee](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/0-1.gif)](assets/demo_videos/0-1.mp4) | [![LaMo milk coffee](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/0-2.gif)](assets/demo_videos/0-2.mp4) |
| Coin spins rapidly on a wooden table. | [![CogVideoX spinning coin](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/1-1.gif)](assets/demo_videos/1-1.mp4) | [![LaMo spinning coin](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/1-2.gif)](assets/demo_videos/1-2.mp4) |
| A honey dipper drizzles honey onto Greek yogurt. | [![CogVideoX honey drizzling](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/2-1.gif)](assets/demo_videos/2-1.mp4) | [![LaMo honey drizzling](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/2-2.gif)](assets/demo_videos/2-2.mp4) |
| An apple falls into a vat of cider, sending up a spray. | [![CogVideoX apple cider splash](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/3-1.gif)](assets/demo_videos/3-1.mp4) | [![LaMo apple cider splash](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/3-2.gif)](assets/demo_videos/3-2.mp4) |
| A pro surfer sails smoothly on the wave-kissed waters. | [![CogVideoX surfer](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/4-1.gif)](assets/demo_videos/4-1.mp4) | [![LaMo surfer](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/4-2.gif)](assets/demo_videos/4-2.mp4) |
| Tablecloth is draped over the dining table. | [![CogVideoX tablecloth](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/5-1.gif)](assets/demo_videos/5-1.mp4) | [![LaMo tablecloth](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/5-2.gif)](assets/demo_videos/5-2.mp4) |
| Pouring beer into a glass, creating white foam. | [![CogVideoX pouring beer](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/6-1.gif)](assets/demo_videos/6-1.mp4) | [![LaMo pouring beer](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/6-2.gif)](assets/demo_videos/6-2.mp4) |
| A mountain biker descends fast through a dirt trail. | [![CogVideoX mountain biker](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/7-1.gif)](assets/demo_videos/7-1.mp4) | [![LaMo mountain biker](https://raw.githubusercontent.com/Applied-Intuition-Open-Source/LaMo/main/assets/demo_gifs/7-2.gif)](assets/demo_videos/7-2.mp4) |

## Main Results

### VideoPhy

SA measures semantic adherence and PC measures physical commonsense.

| Method | Extra Supervision | Overall SA | Overall PC |
|---|---:|---:|---:|
| CogVideoX-2B | - | 60.5 | 25.6 |
| MoAlign-2B (reimpl.) | VideoMAE | 64.5 | 30.1 |
| VideoREPA-2B | VideoMAEv2 | 64.2 | 29.7 |
| **LaMo-2B** | **Self-supervised** | **67.2** | **31.4** |
| CogVideoX-5B | - | 70.0 | 32.3 |
| PhyT2V-5B | o1-preview | 61.0 | 37.0 |
| WISA-5B | Qwen2VL | 67.0 | 38.0 |
| PHANTOM-5B | V-JEPA2 | 47.5 | 37.9 |
| MoAlign-5B (reimpl.) | VideoMAE | 72.2 | 39.4 |
| VideoREPA-5B | VideoMAEv2 | 72.1 | 40.1 |
| **LaMo-5B** | **Self-supervised** | **73.0** | **41.0** |

### VideoPhy2

| Method | SA | PC |
|---|---:|---:|
| CogVideoX-2B | 21.0 | 68.0 |
| PHANTOM-5B | 27.8 | 71.7 |
| MoAlign-2B (paper) | 28.8 | 75.0 |
| MoAlign-2B (reimpl.) | 24.6 | 73.1 |
| VideoREPA-2B | 21.0 | 72.5 |
| **LaMo-2B** | **25.4** | **75.4** |

### VBench

| Method | Quality Score | Semantic Score | Total Score |
|---|---:|---:|---:|
| CogVideoX-5B | 80.5 | 68.7 | 78.2 |
| **LaMo-5B** | **81.9** | **70.7** | **79.6** |

## Interpretability

LaMo's motion prior provides heatmaps that localize physically active regions, including dominant latent drift and prompt-conditioned motion-field responses.

## Getting Started

### Installation

```bash
git clone <LAMO_REPO_URL>
cd <PATH_TO_LAMO_REPO>

conda create -n lamo python=3.10 -y
conda activate lamo

pip install --upgrade pip
pip install -r requirements.txt
pip install -e .
```

Model and evaluator weights are not included. Put them under `pretrain_models/`
or pass local paths to the launchers. See [docs/getting_started.md](docs/getting_started.md)
for the full asset layout and benchmark notes.

### Train

```bash
# Edit train/configs/model_config_train_eval.yaml first.
MODEL_CONFIG_YAML=train/configs/model_config_train_eval.yaml \
bash train/scripts/run_cluster_train_local_lamo.sh
```

### Inference

Generate with LaMo:

```bash
CUDA_VISIBLE_DEVICES=0 python tests/generate_lamo.py \
 --model_path <PATH_TO_COGVIDEOX_BASE_MODEL> \
 --lora_path <PATH_TO_LORA_WEIGHT_DIR> \
 --physical_module_path <PATH_TO_PREDICTOR_SAFETENSORS> \
 --prompt "A bowl of clear water slowly freezes into a transparent block of ice." \
 --output_file outputs/lamo \
 --guidance_lambda 15.0 \
 --guidance_step_ratio 0.8
```

If `predictor.safetensors` is placed next to the LoRA weights,
`--physical_module_path` can be omitted.

### Evaluate

```bash
bash eval/scripts/download_videocon_physics.sh
bash eval/scripts/download_videophy2_auto.sh
bash eval/scripts/download_vbench_pretrained.sh
```

```bash
CUDA_VISIBLE_DEVICES=0,1 bash eval/scripts/run_videophy_eval.sh
CUDA_VISIBLE_DEVICES=0,1 bash eval/scripts/run_videophy2_eval.sh
CUDA_VISIBLE_DEVICES=0,1 bash eval/scripts/run_vbench_eval.sh
```

### Visualize

Run LaMo interpretability visualization:

```bash
CUDA_VISIBLE_DEVICES=0 python tests/interpret_lamo.py \
 --model_path <PATH_TO_COGVIDEOX_BASE_MODEL> \
 --lora_path <PATH_TO_LORA_WEIGHT_DIR> \
 --predictor_ckpt <PATH_TO_PREDICTOR_SAFETENSORS> \
 --prompt "A coin spins rapidly on a wooden table." \
 --num_samples 1 \
 --work_dir outputs/interpret_lamo
```

The script writes per-sample frames, BMV heatmaps, predictor-response heatmaps, a combined grid, and raw arrays under `--work_dir`.

For full setup and benchmark details, see [docs/getting_started.md](docs/getting_started.md).

## Repository Layout

- `finetrainers/`: LaMo model, trainer, processors, and utility code.
- `train/`: training entry points, launchers, accelerate configs, and train-eval config templates.
- `eval/`: benchmark launchers, eval configs, metadata, and evaluator code for VideoPhy, VideoPhy2, and VBench.
- `docs/`: extended setup, training, inference, and benchmark notes.
- `tests/generate_lamo.py`: LaMo text-to-video inference.
- `tests/generate_cogvideox.py`: CogVideoX baseline inference.
- `tests/interpret_lamo.py`: LaMo interpretability visualization.

## Citation

If you find LaMo useful, please consider citing:

```bibtex
@article{jiang2026lamo,
  title={LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation},
  author={Bo Jiang and Depu Meng and Yihan Hu and Yichen Xie and Tianshuo Xu and Wei Zhan},
  journal={arXiv preprint arXiv:2605.23878},
  year={2026}
}
```

## Acknowledgements

This work was conducted at Applied Intuition. LaMo builds upon
[WISA](https://github.com/360CVGroup/WISA),
[VideoREPA](https://github.com/aHapBean/VideoREPA), and
[Diffusers](https://github.com/huggingface/diffusers). We thank the authors and
maintainers for releasing their code.

## License

This repository vendors and adapts third-party code (finetrainers, VBench,
VideoPhy, mPLUG-Owl); see [NOTICE](NOTICE) for attributions and their
license terms.

LaMo

오너의 다른 레포

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

LaMo

오너의 다른 레포

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README