AFB-URR - GitLite

KO | EN

gitlite — search

#javascript #python #react #typescript #go #golang #android #macos #java #kubernetes #awesome-list #pytorch #ios #open-source

AFB-URR

(★ 105)

Official implementation of "Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement", NeurIPS 2020.

File Explorer

Download Latest Version (.zip)

- afb-urr-compressed.gif
- mask_palette.png
- pipeline.png
- pretrain.txt
- __init__.py
- DAVIS_DS.py
- LongVideo_DS.py
- PreTrain_DS.py
- transforms.py
- YouTube_DS.py
- - metrics.py
  - utils.py
- dataset.py
- evaluation.py
- README.md
- results.py
- score_longvideo.py
- __init__.py
- AFB_URR.py
- FeatureBank.py
- __init__.py
- data.py
- system.py
.gitignore
eval.py
README.md
requirements.txt
train.py
unify_pretrain_dataset.py

# Installation Guide

1. Get the code

git clone https://github.com/xmlyqing00/AFB-URR

Downloads the entire project code from GitHub to your computer.

cd AFB-URR

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip3 install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement

This repository is the official implementation of Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement (NeurIPS 2020).
It is designed for semi-supervised video object segmentation (VOS) task.

[\[NeurIPS Page\]](https://proceedings.neurips.cc//paper_files/paper/2020/hash/234833147b97bb6aed53a8f4f1c7a7d8-Abstract.html)
[\[Paper\]](https://proceedings.neurips.cc/paper/2020/file/234833147b97bb6aed53a8f4f1c7a7d8-Paper.pdf)
[\[Supplementary\]](https://proceedings.neurips.cc/paper/2020/file/234833147b97bb6aed53a8f4f1c7a7d8-Supplemental.zip)

![](https://raw.githubusercontent.com/xmlyqing00/AFB-URR/main/assets/afb-urr-compressed.gif)

![](https://raw.githubusercontent.com/xmlyqing00/AFB-URR/main/assets/pipeline.png)

**Paper corrections:** Our feature map generated by the encoders has 1024 channels and **1/16** of the original image size.

## 1. Requirements

We built and tested the repository on Python 3.6.9 and Ubuntu 18.04 with one NVIDIA 1080Ti card (11GB Memory).
Run on Windows or Mac is possible with minor modifications. 
An NVIDIA GPU card and CUDA environment are required.
To install requirements, run:
```bash
pip3 install -r requirements.txt
```

Install the package [torch_scatter](https://github.com/rusty1s/pytorch_scatter) by the official instructions. Our version is 2.0.4.

## 2. Evaluation

### DAVIS17-TrainVal
1. Download and extract [DAVIS17-TrainVal](https://data.vision.ee.ethz.ch/csergi/share/davis/DAVIS-2017-trainval-480p.zip) dataset.
2. Download the pretrained DAVIS17 [checkpoint](https://huggingface.co/xmlyqing00/AFB-URR/tree/main).
3. run:
```bash
python3 eval.py --level 1 --resume /path/to/checkpoint.pth/ --dataset /path/to/dir/
```
To reproduce the segmentation scores, you can use the official evaluation [tool](https://davischallenge.org/davis2017/code.html) from DAVIS benchmark.

### YouTube-VOS18
1. Download and extract [YouTube-VOS18](https://competitions.codalab.org/competitions/19544#participate-get_data) dataset.
2. Download the pretrained YouTube-VOS18 [checkpoint](https://huggingface.co/xmlyqing00/AFB-URR/tree/main).
3. run:
```bash
python3 eval.py --level 2 --resume /path/to/checkpoint.pth/ --dataset /path/to/dir/ --update-rate 0.05
```
**Attention**: Directly submit our results to the YouTube-VOS codalab for evaluation will pollute the leader board.
We encourage you to submit your own results.

### Long Videos
1. Download and extract [Long Videos](https://www.kaggle.com/gvclsu/long-videos) dataset.
2. Download the pretrained YouTube-VOS18 checkpoint above.
3. run:
```bash
python3 eval.py --level 3 --resume /path/to/checkpoint.pth/ --dataset /path/to/dir/ --update-rate 0.05
```
To reproduce the segmentation scores, you can use the same tool from the DAVIS benchmark.

### Your Own Video

Prepare your video frames and the first frame annotation following the data structure of the long videos page.
You can see the data structure without download it and you only need to provide the first frame annotation.

Run the same parameters as the long videos setting.

### Options for Evaluation
1. `--gpu`: GPU id to run (default: 0).
2. `--viz`: Enable output overlays along with the estimated masks (default: False).
3. `--budget`: The number of features that can be stored in total (default: 300000 for 1080Ti).

By default, the segmentation results will be saved in `./output`.

## 3. Training

### Pre-training on Static Images

1. Download the following the datasets (COCO is the largest one). You don't have to download all, our pretrain codes skip datasets that don't exist by default.
2. Run `unify_pretrain_dataset.py` to convert them into a uniform format (followed DAVIS).
```bash
python3 unify_pretrain_dataset.py --name NAME --src /path/to/dataset/dir/ --dst /path/to/output
```
1. [MSRA10K](https://mmcheng.net/msra10k/):  `--name MSRA10K`
2. [ECSSD](http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency/dataset.html): `--name ECSSD`
3. [PASCAL-S](http://cbs.ic.gatech.edu/salobj/download/salObj.zip): `--name PASCAl-s`
4. [PASCAL VOC2012](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/): `--name PASCALVOC2012`
5. [COCO](http://cocodataset.org/#download): `--name COCO`.
    API [pycocotools](https://github.com/cocodataset/cocoapi/tree/master/PythonAPI) is required.

You may need minor modifications in the dataset path. Descriptions of useful options,
1. `--palette`: Path to the palette image. We provide a template in `assets/mask_palette.png`, followed the formats of DAVIS17.
2. `--workder`: The parallel threads number to accelerate the procedures (Default: 20).

After the conversion process, you can start pre-training the model:
```bash
python3 train.py --level 0 --dataset /path/to/pretrain/ --lr 1e-5 --scheduler-step 3 --total-epoch 12 --log
```  
Pre-training process may takes days to weeks, you can download our [checkpoint](https://huggingface.co/xmlyqing00/AFB-URR/tree/main) to save time.

### Training on DAVIS17

Download the semi-supervised TrainVal 480p from the [DAVIS](https://davischallenge.org/davis2017/code.html) website.
Run 
```bash
python3 train.py --level 1 --new --resume /path/to/PreTrain/checkpoint.pth --dataset /path/to/DAVIS17/ --lr 4e-6 --scheduler-step 200 --total-epoch 1000 --log
```

### Training on YouTube-VOS
Download training set of the [YouTube-VOS](https://competitions.codalab.org/competitions/19544#participate-get_data) dataset.
Run
```bash
python3 train.py --level 2 --new --resume /path/to/PreTrain/checkpoint.pth --dataset /path/to/YouTubeVOS/train --lr 4e-6 --scheduler-step 30 --total-epoch 150 --log
```

## 4. License

This repository is released for academic use only. If you want to use our codes for commercial products, please contact xinli@cct.lsu.edu in advance.
If you use our codes, please cite our paper,
```bibtex
@inproceedings{NEURIPS2020_liangVOS,
 author = {Liang, Yongqing and Li, Xin and Jafari, Navid and Chen, Jim},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {3430--3441},
 publisher = {Curran Associates, Inc.},
 title = {Video Object Segmentation with Adaptive Feature Bank and Uncertain-Region Refinement},
 url = {https://proceedings.neurips.cc/paper/2020/file/234833147b97bb6aed53a8f4f1c7a7d8-Paper.pdf},
 volume = {33},
 year = {2020}
}
```

## 5. Update Logs
- 2022/04/24 Update the evaluation script for long video benchmark.