Be-Your-Outpainter - GitLite

KO | EN

gitlite — search

#cpp #nodejs #developer-tools #privacy #linux #sql #pytorch #awesome #gpt #codex

Be-Your-Outpainter

(★ 255)

[ECCV 2024] Be-Your-Outpainter https://arxiv.org/abs/2403.13745

파일 탐색기

ZIP 다운로드

- base-lora.yaml
- base.yaml
- exp-long.yaml
- exp.yaml
- - dirt-road-driving.mp4
  - SB_Bear.mp4
  - SB_Dog1.mp4
  - SB_Eagle.mp4
  - SB_Portrait1.mp4
  - SB_Portrait2.mp4
  - SB_Running4.mp4
- train_outpaint.py
- train_outpaint_long.py
- - naive_dataset.py
- - pipelineoutpaint.py
- - scheduling_ddim.py
- - lora.py
  - metrics.py
  - register.py
  - register_moe.py
- util.py
environment.yml
LICENSE
README.md
run.sh
run_long.sh

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`conda env`

설명 보기 ▼

Manage conda environments.

conda env create {{[-f|--file]}} {{path/to/file}}

Create an environment from an environment file (YAML, TXT, etc.):

conda env remove {{[-n|--name]}} {{environment_name}}

Delete an environment and everything in it:

conda env update {{[-f|--file]}} {{path/to/file}} --prune

Update an environment based on an environment file:

🔍

`git clone`

설명 보기 ▼

Clone an existing repository.

git clone {{remote_repository_location}} {{path/to/directory}}

Clone an existing repository into a new directory (the default directory is the repository name):

git clone --recursive {{remote_repository_location}}

Clone an existing repository and its submodules:

git clone {{[-n|--no-checkout]}} {{remote_repository_location}}

Clone only the `.git` directory of an existing repository:

# 프로젝트 배지

diversity

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

https://github.com/G-U-N/Be-Your-Outpainter/assets/60997859/9c10fde7-c881-4dd6-89b3-00b95ef9d099

### Run

1. Install Environment

```shell
conda env create -f environment.yml
```
2. Downloads

Download the models folder from Huggingface.

```shell
git clone https://huggingface.co/wangfuyun/Be-Your-Outpainter
```

3. Run the code for basic testing. Single GPU with 20GB memory is required for current code version. Reduce the video length if GPU memory is limited.

```shell
bash run.sh
```
Check the outpainted results from the `results' folder.

### Outpaint Your Own Videos
Edit the [exp.yaml](https://github.com/G-U-N/Be-Your-Outpainter/blob/master/configs/exp.yaml) to outpaint your own videos.
```yaml
exp: # Name of your task

train_data:
    video_path: "data/outpaint_videos/SB_Dog1.mp4"                            # source video path
    prompt: "a cute dog, garden, flowers"                                     # source video prompts for tuning
    n_sample_frames: 16                                                       # source video length
    width: 256                                                                # source video width
    height: 256                                                               # source video height
    sample_start_idx: 0                                                       # set to 0 by default. Sampling frames from the beginning of the video
    sample_frame_rate: 1                                                      # fps of video 
  
  validation_data:
    prompts:
      - "a cute dog, garden, flowers"                                         # prompts applied for outpainting. 
    prompts_l:
      - "wall"
    prompts_r:
      - "wall"
    prompts_t:
      - ""
    prompts_b:
      - ""

prompts_neg:
      - ""

is_grid: False                                                            # set as True to enable prompts_r, prompts_l, prompts_t, prompts_b 
    video_length: 16                                                          # video length. The same as in the train_data config
    width: 256
    height: 256

scale_l: 0
    scale_r: 0
    scale_t: 0.5                                                              # How to expand the video field. For a 512x512 source video. Set scale_l and scale_r to 0.5, and it will generate 512x(512 + 512 * 0.5 + 512 * 0.5) = 512 x 1024 video.
    scale_b: 0.5

window_size: 16                                                           # only used in longer video outpainting
    stride: 4

repeat_time: 0                                                            # set to 4 enable noise regret
    jump_length: 3

num_inference_steps: 50                                                   # inference steps for outpainting
    guidance_scale: 7.5

bwd_mask: null                                                            # not applied
    fwd_mask: null
    bwd_flow: null
    fwd_flow: null

warp_step: [0,0.5]
    warp_time: 3

mask_config:                                                                # how to set mask for tuning
    mask_l: [0., 0.4]
    mask_r: [0., 0.4]
    mask_t: [0., 0.4]
    mask_b: [0., 0.4]
```

### Evaluation
Please check [metric.py](https://github.com/G-U-N/Be-Your-Outpainter/blob/master/src/utils/metrics.py).

Important notes:
- The metrics are reported as the average value of the settings with mask ratio 0.25 and mask ratio 0.66. For example, if the PSNR of mask ratio 0.25 is 30 and the PSNR of the mask ratio 0.66 is 20, and then the reported PSNR is (20+30)/2 = 25.
- The VAE might suffer from decoding losses. We replace the known region with the GT for evaluation.

### Cite
```bibtex
@article{wang2024your,
  title={Be-Your-Outpainter: Mastering Video Outpainting through Input-Specific Adaptation},
  author={Wang, Fu-Yun and Wu, Xiaoshi and Huang, Zhaoyang and Shi, Xiaoyu and Shen, Dazhong and Song, Guanglu and Liu, Yu and Li, Hongsheng},
  journal={arXiv preprint arXiv:2403.13745},
  year={2024}
}
```