VFDepth - GitLite

KO | EN

gitlite — search

#react #ai #llm #golang #go #machine-learning #rust #deep-learning #docker #nodejs #java #security #ios #open-source

VFDepth

(★ 108)

VFDepth Self-supervised surround-view depth estimation with volumetric feature fusion

파일 탐색기

최종 버전 다운로드 (.zip)

- - ddad_baseline.yaml
  - ddad_baseline_ddp.yaml
  - ddad_surround_fusion.yaml
  - ddad_surround_fusion_augdepth.yaml
  - ddad_surround_fusion_ddp.yaml
- - nusc_baseline.yaml
  - nusc_baseline_ddp.yaml
  - nusc_surround_fusion.yaml
  - nusc_surround_fusion_ddp.yaml
- - mask_idx_dict.pkl
- - train.txt
  - val.txt
- - CAM_BACK_LEFT_mask.png
  - CAM_BACK_mask.png
  - CAM_BACK_RIGHT_mask.png
  - CAM_FRONT_LEFT_mask.png
  - CAM_FRONT_mask.png
  - CAM_FRONT_RIGHT_mask.png
- __init__.py
- base_dataset.py
- data_util.py
- ddad_dataset_sf.py
- nuscenes_dataset.py
- - __init__.py
- - __init__.py
- - __init__.py
- dgp
- packnet_sfm
- depth_synthesis.gif
- surround_fusion_feature.gif
- - __init__.py
  - geometry_util.py
  - pose.py
  - view_rendering.py
- - __init__.py
  - base_loss.py
  - depth_synthesis_loss.py
  - loss_util.py
  - multi_cam_loss.py
  - single_cam_loss.py
- __init__.py
- base_model.py
- vfdepth.py
- __init__.py
- blocks.py
- fusion_depthnet.py
- fusion_posenet.py
- mono_depthnet.py
- mono_posenet.py
- volumetric_fusionnet.py
- __init__.py
- vfdepth_trainer.py
- __init__.py
- ddp.py
- logger.py
- misc.py
- visualize.py
.gitignore
.gitmodules
CODE_OF_CONDUCT.md
eval.py
LICENSE
README.md
requirements.txt
train.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/42dot/VFDepth

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd VFDepth

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Self-supervised surround-view depth estimation with volumetric feature fusion 
Jung-Hee Kim\*, Junwha Hur\*, Tien Nguyen, and Seong-Gyun Jeong - NeurIPS 2022 \
Link to the paper: [Link](https://openreview.net/forum?id=0PfIQs-ttQQ&referrer=%5BAuthor%20Console%5D(%2Fgroup%3Fid%3DNeurIPS.cc%2F2022%2FConference%2FAuthors%23your-submissions)) 
```
@inproceedings{kimself,
  title={Self-supervised surround-view depth estimation with volumetric feature fusion},
  author={Kim, Jung Hee and Hur, Junhwa and Nguyen, Tien Phuoc and Jeong, Seong-Gyun},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)}
  year = {2022},
}
```
## Introduction
We introduce a volumetric feature representation for self-supervised surround-view depth approach, which not only outputs metric-scale depth and canonical camera motion, but also synthesizes a depth map at a novel viewpoint. \
 <img 
 width="600" 
 height="356" 
 src="media/surround_fusion_feature.gif" 
 />
 <br>
 <img  
 width="600" 
 height="356" 
 src="media/depth_synthesis.gif" 
 />

## Installation
* Install required libraries using the  `requirements.txt` file. \
(Note that we leverage `packent-sfm`, `dgp` as submodules and therefore need to install required libraries related to the submodules.) 
* To install both required library and submodules, you need to follow instruction below:

```shell
git submodule init
git submodule update
pip install -r requirements.txt
```

## Datasets
### DDAD 
* DDAD dataset can be downloaded by running:

```shell
curl -s https://tri-ml-public.s3.amazonaws.com/github/DDAD/datasets/DDAD.tar 
```

* Place the dataset in `input_data/DDAD/`
* We manually created mask image for scene of ddad dataset and are provided in `dataset/ddad_mask`

### NuScenes 
* Download NuScenes official dataset
* Place the dataset in `input_data/nuscenes/`
* Scenes with backward and forward contexts are listed in `dataset/nuscenes/`
* Scenes with low visibility are filtered in `dataset/nuscenes/val.txt`

Data should be as follows:
```
├── input_data
│   ├── DDAD
│   │   ├── ddad_train_val
│   │   ├── ddad_test
│   ├── nuscenes
│   │   ├── maps
│   │   ├── samples
│   │   ├── sweeps
│   │   ├── v1.0-test
|   |   ├── v1.0-trainval
```
## Main Results

<table>
  <tr>
    <td>Model</td>
    <td>Scale</td>
    <td>Abs.Rel.</td>
    <td>Sq.Rel.</td>
    <td>RMSE</td>
    <td>RMSElog</td>
    <td>d<sub>1.25</sub></td>
    <td>d<sub>1.25</sub><sup>2</sup></td>
    <td>d<sub>1.25</sub><sup>3</sup></td>
  </tr>
  <tr>
    <td rowspan="2"><a href="https://drive.google.com/drive/folders/1UAPjm3pplVPkbS6jLsIFWDmsR8O8dX-1?usp=drive_link"> DDAD </a> </td>
    <td style="text-align:left">Metric</td>
    <td style="text-align:center">0.221</td>
    <td style="text-align:center">4.001</td>
    <td style="text-align:center">13.406</td>
    <td style="text-align:center">0.340</td>
    <td style="text-align:center">0.688</td>
    <td style="text-align:center">0.868</td>
    <td style="text-align:center">0.932</td>
  </tr>
  <tr>
    <td style="text-align:left">Median</td>
    <td style="text-align:center">0.221</td>
    <td style="text-align:center">3.884</td>
    <td style="text-align:center">13.225</td>
    <td style="text-align:center">0.328</td>
    <td style="text-align:center">0.692</td>
    <td style="text-align:center">0.877</td>
    <td style="text-align:center">0.939</td>
  </tr>
  <tr>
    <td rowspan="2"><a href="https://drive.google.com/drive/folders/1CipDGyLCYZcWKQSo1WlLbGsyn2ssL8dI?usp=drive_link"> NuScenes </a> </td>
    <td style="text-align:left">Metric</td>
    <td style="text-align:center">0.285</td>
    <td style="text-align:center">6.662</td>
    <td style="text-align:center">7.472</td>
    <td style="text-align:center">0.347</td>
    <td style="text-align:center">0.741</td>
    <td style="text-align:center">0.883</td>
    <td style="text-align:center">0.936</td>
  </tr>
  <tr>
    <td style="text-align:left">Median</td>
    <td style="text-align:center">0.258</td>
    <td style="text-align:center">4.282</td>
    <td style="text-align:center">7.226</td>
    <td style="text-align:center">0.329</td>
    <td style="text-align:center">0.735</td>
    <td style="text-align:center">0.883</td>
    <td style="text-align:center">0.937</td>
  </tr>  
</table>

## Get Started
### Training
Surround-view fusion depth estimation model can be trained from scratch.
* By default results are saved under `results/<config-name>` with trained model and tensorboard file for both training and validation.

**Single-GPU** <br>
Training the model using single-GPU: \
(Note that, due to usage of packnet-sfm submodule, userwarning repetitively occurs and therefore ignored while training.) 
```shell
python -W ignore train.py --config_file='./configs/ddad/ddad_surround_fusion.yaml'
python -W ignore train.py --config_file='./configs/nuscenes/nusc_surround_fusion.yaml'
```

**Multi-GPU** <br>
Training the model using Multi-GPU:
* Enable distributed data parallel(DDP), by setting ddp:ddp_enable to True in the config file
  * Gpus and the worldsize(number of gpus) must be specified (ex. gpus = [0, 1, 2, 3], worldsize= 4)
* DDP address and port setting can be configured in [ddp.py](utils/ddp.py)

```shell
python -W ignore train.py --config_file='./configs/ddad/ddad_surround_fusion_ddp.yaml' 
python -W ignore train.py --config_file='./configs/nuscenes/nusc_surround_fusion_ddp.yaml'
```

### Evaluation
To evaluate the trained model from scratch, run:
```shell
python -W ignore eval.py --config_file='./configs/<config-name>'
```
* The model weights need to be specified in `load: weights` of the config file.

Evaluation results using the pretrained model can be obtained by using the following command:
```shell
python -W ignore eval.py --config_file='./configs/<config-name>' \
                         --weight_path='<pretrained-weight-path>'
```

### Depth Synthesis
To obtain synthesized depth results, train the model from scratch by running: 
```shell
python -W ignore train.py --config_file='./configs/ddad/ddad_surround_fusion_augdepth.yaml'
```

Then evaluate the model by running:
```shell
python -W ignore eval.py --config_file='./configs/ddad/ddad_surround_fusion_augdepth.yaml'
```

* The synthesized results are stored `results/<config-name>/syn_results`

## License
This repository is released under the [Apach 2.0](LICENSE) license.