Ha-NeRF - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #typescript #llm #golang #rust #deep-learning #linux #java #windows #kubernetes #vue #openai

Ha-NeRF

(★ 160)

[CVPR 2022] Ha-NeRF😆: Hallucinated Neural Radiance Fields in the Wild

파일 탐색기

최종 버전 다운로드 (.zip)

- 1.jpeg
- 2.jpg
- 3.jpg
- 4.jpg
- 5.jpg
- 6.jpg
- __init__.py
- blender_mask_grid_sample.py
- colmap_utils.py
- depth_utils.py
- global_val.py
- phototourism_mask_grid_sample.py
- PhototourismDataset.py
- ray_utils.py
- __init__.py
- nerf.py
- networks.py
- rendering.py
- __init__.py
- save_weights_only.py
- visualization.py
- warmup_scheduler.py
eval.py
eval_metric.py
hallucinate.py
losses.py
metrics.py
opt.py
perturbation_visualization.py
prepare_phototourism.py
README.md
requirements.txt
train_mask_grid_sample.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/rover-xingyu/Ha-NeRF

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd Ha-NeRF

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

* Install core requirements by `pip install -r requirements.txt`

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Ha-NeRF:laughing:: Hallucinated Neural Radiance Fields in the Wild 
**[Project Page](https://rover-xingyu.github.io/Ha-NeRF/) |
[Paper](https://openaccess.thecvf.com/content/CVPR2022/papers/Chen_Hallucinated_Neural_Radiance_Fields_in_the_Wild_CVPR_2022_paper.pdf) |
[Latest arXiv](https://arxiv.org/pdf/2111.15246.pdf) |
[Supplementary](https://rover-xingyu.github.io/Ha-NeRF/files/Ha_NeRF_CVPR_2022_supp.pdf)**

[Xingyu Chen¹](https://scholar.google.com/citations?user=gDHPrWEAAAAJ&hl=en), 
[Qi Zhang²](https://scholar.google.com/citations?user=2vFjhHMAAAAJ&hl=en), 
[Xiaoyu Li²](https://scholar.google.com/citations?user=Dt0PcAYAAAAJ&hl=en), 
[Yue Chen¹](https://scholar.google.com/citations?user=M2hq1_UAAAAJ&hl=en), 
[Ying Feng²](https://scholar.google.com/citations?user=PhkrqioAAAAJ&hl=en),
[Xuan Wang²](https://scholar.google.com/citations?user=h-3xd3EAAAAJ&hl=en),
[Jue Wang²](https://scholar.google.com/citations?user=Bt4uDWMAAAAJ&hl=en).

[¹Xi'an Jiaotong University](http://en.xjtu.edu.cn/),
[²Tencent AI Lab](https://ai.tencent.com/ailab/en/index/).

This repository is an official implementation of [Ha-NeRF](https://rover-xingyu.github.io/Ha-NeRF/) (Hallucinated Neural Radiance Fields in the Wild) using pytorch ([pytorch-lightning](https://github.com/PyTorchLightning/pytorch-lightning)).

# :computer: Installation

## Hardware

* OS: Ubuntu 18.04
* NVIDIA GPU with **CUDA>=10.2**
* We optimize all implementations for 600 000 iterations with a batch size of 1024 on 4 A100s, where Ha-NeRF and NeRF-W take 20 and 18 hours, respectively. Both methods require around 20 GB of memory for training and 5 GB for inference.

## Software

* Clone this repo by `git clone https://github.com/rover-xingyu/Ha-NeRF`
* Python>=3.6 (installation via [anaconda](https://www.anaconda.com/distribution/) is recommended, use `conda create -n HaNeRF python=3.6` to create a conda environment and activate it by `conda activate HaNeRF`)
* Python libraries
    * Install core requirements by `pip install -r requirements.txt`
    
# :key: Training

## Data download

Download the scenes you want from [here](https://www.cs.ubc.ca/~kmyi/imw2020/data.html)

Download the train/test split from [here](https://nerf-w.github.io/) and put under each scene's folder (the **same level** as the "dense" folder)

(Optional but **highly** recommended) Run `python3.6 prepare_phototourism.py --root_dir $ROOT_DIR --img_downscale {an integer, e.g. 2 means half the image sizes}` to prepare the training data and save to disk first, if you want to run multiple experiments or run on multiple gpus. This will **largely** reduce the data preparation step before training.

Run (example)

```
python3.6 prepare_phototourism.py --root_dir /path/to/the/datasets/brandenburg_gate/ --img_downscale 2
```

## Training model
Run (example)
```
python3.6 train_mask_grid_sample.py \
  --root_dir /path/to/the/datasets/brandenburg_gate/ --dataset_name phototourism \
  --save_dir save \
  --img_downscale 2 --use_cache \
  --N_importance 64 --N_samples 64 \
  --num_epochs 20 --batch_size 1024 \
  --optimizer adam --lr 5e-4 --lr_scheduler cosine \
  --exp_name exp_HaNeRF_Brandenburg_Gate \
  --N_emb_xyz 15 --N_vocab 1500 \
  --use_mask --maskrs_max 5e-2 --maskrs_min 6e-3 --maskrs_k 1e-3 --maskrd 0 \
  --encode_a --N_a 48 --weightKL 1e-5 --encode_random --weightRecA 1e-3 --weightMS 1e-6 \
  --num_gpus 4
```

Add `--encode_a` for using appearance hallucination module, `--use_mask` for using  anti-occlusion module. `--N_vocab` should be set to an integer larger than the number of images (dependent on different scenes). For example, "brandenburg_gate" has in total 1363 images (under `dense/images/`), so any number larger than 1363 works (no need to set to exactly the same number). **Attention!** If you forget to set this number, or it is set smaller than the number of images, the program will yield `RuntimeError: CUDA error: device-side assert triggered` (which comes from `torch.nn.Embedding`).

See [opt.py](opt.py) for all configurations.

The checkpoints and logs will be saved to `{save_dir}/ckpts/{scene_name} ` and `{save_dir}/logs/{scene_name}`, respectively.

You can monitor the training process by `tensorboard --logdir save/logs/exp_HaNeRF_Brandenburg_Gate --port=8600` and go to `localhost:8600` in your browser.

# :mag_right: Evaluation

Use [eval.py](eval.py) to inference on all test data. It will create folder `{save_dir}/results/{dataset_name}/{scene_name}` and save the rendered
images.

Run (example)
```
python3.6 eval.py \
  --root_dir /path/to/the/datasets/brandenburg_gate/ \
  --save_dir save \
  --dataset_name phototourism --scene_name HaNeRF_Trevi_Fountain \
  --split test_test --img_downscale 2 \
  --N_samples 256 --N_importance 256 --N_emb_xyz 15 \
  --N_vocab 1500 --encode_a \
  --ckpt_path save/ckpts/HaNeRF_Brandenburg_Gate/epoch\=19.ckpt \
  --chunk 16384 --img_wh 320 240
```

Then you can use [eval_metric.py](eval_metric.py) to get the quantitative report of different metrics based on the rendered images from [eval.py](eval.py). It will create a file `result.txt` in the folder `{save_dir}/results/{dataset_name}/{scene_name}` and save the metrics.

Run (example)
```
python3.6 eval_metric.py \
  --root_dir /path/to/the/datasets/brandenburg_gate/ \
  --save_dir save \
  --dataset_name phototourism --scene_name HaNeRF_Trevi_Fountain \
  --split test_test --img_downscale 2 \
  --img_wh 320 240
```

# :laughing: Hallucination

Use [hallucinate.py](hallucinate.py) to play with Ha-NeRF by hallucinating appearance from different scenes `{example_image}` in different views! It will create folder `{save_dir}/hallucination/{scene_name}` and render the hallucinations, finally create a gif out of them.

Run (example)
```
python3.6 hallucinate.py \
    --save_dir save \
    --ckpt_path save/ckpts/HaNeRF_Trevi_Fountain/epoch\=19.ckpt \
    --chunk 16384 \
    --example_image artworks \
    --scene_name artworks_2_fountain
```

# Cite
If you find our work useful, please consider citing:
```bibtex
@inproceedings{chen2022hallucinated,
  title={Hallucinated neural radiance fields in the wild},
  author={Chen, Xingyu and Zhang, Qi and Li, Xiaoyu and Chen, Yue and Feng, Ying and Wang, Xuan and Wang, Jue},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={12943--12952},
  year={2022}
}
```

# Acknowledge
Our code is based on the awesome pytorch implementation of NeRF in the Wild ([NeRF-W](https://github.com/kwea123/nerf_pl/tree/nerfw/)). We appreciate all the contributors.