Birds-eye-view-Perception - GitLite

KO | EN

gitlite — search

#ai-agents #editor #openai #github #cpp #pdf #cli #hacktoberfest #anthropic #java

Birds-eye-view-Perception

(★ 1,381)

[IEEE T-PAMI 2023] Awesome BEV perception research and cookbook for all level audience in autonomous diriving

File Explorer

ZIP Download

- FUNDING.yml
- - __init__.py
  - functional.py
  - transforms.py
- - __init__.py
  - data_aug_mmdet3d.py
- __init__.py
- init_toolbox.py
- README.md
- - academia.md
  - bev_camera.md
  - bev_fusion.md
  - bev_lidar.md
  - camera_detection.md
  - dataset.md
  - industry.md
  - lidar_detection.md
  - lidar_segmentation.md
  - sensor_fusion.md
- data_preparation.md
- installation.md
- waymo_setup.md
- cam0_img.jpg
- cam1_img.jpg
- cam2_img.jpg
- cam3_img.jpg
- cam4_img.jpg
- example.md
- flip.png
- cam_ablation.jpg
- cam_pipeline_v2.jpg
- dataset_table.jpg
- general_overview.png
- important_methods.jpg
- important_methods_leaderboard.jpg
- industrial_pipeline.jpg
- lidar_ablation.jpg
- lidar_pipeline.jpg
- waymo_coordinate.png
- - .gitignore
  - LICENSE
  - README.md
- - README.md
  - setup.cfg
  - setup.py
- GAPretrain.md
- - __init__.py
- - __init__.py
  - create_data.py
  - dist_test_video.sh
  - dist_train_video.sh
  - test_video.py
  - train_video.py
  - visual.ipynb
.gitignore
.gitmodules
CITATION.cff
CODE_OF_CONDUCT.md
LICENSE
README.md
setup.cfg
setup.py

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

> [!IMPORTANT]
> 🌟 Stay up to date at [opendrivelab.com](https://opendrivelab.com/#news)!

Awesome BEV perception papers and toolbox for achieving state-of-the-arts performance.

### Table of Contents
- [Introduction](#introduction)
  - [Major Features](#major-features)
- [What's New](#whats-new)
- [BEV Algorithm Family](#bev-algorithm-family)
- [BEV Toolbox](#bev-toolbox)
- [Literature Survey](#literature-survey)
- [License and Citation](#license-and-citation)

## Introduction

This repo is associated with the survey paper "[Delving into the Devils of Bird’s-eye-view Perception: A Review, Evaluation and Recipe](https://ieeexplore.ieee.org/document/10321736)", which provides an up-to-date literature survey for BEV perception and an open-source BEV toolbox based on PyTorch. We also introduce the BEV algorithm family, including follow-up work on BEV percepton such as [VCD](https://arxiv.org/abs/2310.15670), [GAPretrain](https://arxiv.org/abs/2304.03105), and [FocalDistiller](https://openaccess.thecvf.com/content/CVPR2023/html/Zeng_Distilling_Focal_Knowledge_From_Imperfect_Expert_for_3D_Object_Detection_CVPR_2023_paper.html). We hope this repo can not only be a good starting point for new beginners but also help current researchers in the BEV perception community.

`If you find some work popular enough to be cited below, email us or simply open a PR!`

### Major Features

* **Up-to-date Literature Survey for BEV Perception** <br> We summarize important methods in recent years about BEV perception, including different modalities (camera, LIDAR, Fusion) and tasks (Detection, Segmentation, Occupancy). More details of the survey paper list can be found [here](./docs/paper_list/).

* **Convenient BEVPerception Toolbox** <br> We integrate a bag of tricks in the BEV toolbox that helps us achieve 1st in the camera-based detection track of the Waymo Open Challenge 2022, which can be used independently or as a plug-in for popular deep learning libraries. Moreover, we provide a suitable playground for beginners in this area, including a hands-on tutorial and a small-scale dataset (1/5 WOD in kitti format) to validate ideas. More details can be found [here](./bev_toolbox/README.md).

* **SOTA BEV Knowledge Distillation Algorithms** <br> We include important follow-up works of BEVFormer/BEVDepth/SOLOFusion in the perspective of knowledge distillation([VCD](https://arxiv.org/abs/2310.15670), [GAPretrain](https://arxiv.org/abs/2304.03105), [FocalDistiller](https://openaccess.thecvf.com/content/CVPR2023/html/Zeng_Distilling_Focal_Knowledge_From_Imperfect_Expert_for_3D_Object_Detection_CVPR_2023_paper.html)). More details of each paper can be found in each README.md file under [here](./nuScenes_playground/).

| Method | Expert | Apprentice |
| :---: | :---: | :---: |
| VCD | Vision-centric multi-modal detector | Camera-only detector |
| GAPretrain  | Lidar-only detector  | Camera-only detector  |
| FocalDistiller   | Camera-only detector  | Camera-only detector  |

## What's New
```2023/11/04``` Our Survey is accepted by IEEE T-PAMI.

```2023/10/26``` A new paper [VCD](https://arxiv.org/abs/2310.15670) is coming soon with official implementation.

```2023/09/06``` We have a new version of the [survey](https://ieeexplore.ieee.org/document/10321736). Check it out!

```2023/04/06``` Two new papers [GAPretrain](https://arxiv.org/abs/2304.03105) and [FocalDistiller](https://openaccess.thecvf.com/content/CVPR2023/html/Zeng_Distilling_Focal_Knowledge_From_Imperfect_Expert_for_3D_Object_Detection_CVPR_2023_paper.html) are coming soon with official implementation.

```2022/10/13``` v0.1 was released.
* Integrate some practical data augmentation methods for BEV camera-based 3D detection in the toolbox.
* Offer a pipeline to process the Waymo dataset (camera-based 3D detection).
* Release a baseline (with config) for the Waymo dataset and also 1/5 of the Waymo dataset in Kitti format.

## Literature Survey

![](https://raw.githubusercontent.com/OpenDriveLab/Birds-eye-view-Perception/master/figs/general_overview.png)
The general picture of BEV perception at a glance, where consists of three sub-parts based on the input modality. BEV perception is a general task built on top of a series of fundamental tasks. For better completeness of the whole perception algorithms in autonomous driving, we list other topics as well. More details can be found in the [survey paper](https://ieeexplore.ieee.org/document/10321736).

We have summarized important datasets and methods in recent years about BEV perception in academia and also different roadmaps used in industry. 
* [Academic Summary of BEV Perception](docs/paper_list/academia.md)
  * [Datasets for BEV Perception](docs/paper_list/dataset.md)
  * [BEV Camera](docs/paper_list/bev_camera.md)
  * [BEV Lidar](docs/paper_list/bev_lidar.md)
  * [BEV Fusion](docs/paper_list/bev_fusion.md)
* [Industrial Roadmap of BEV Perception](docs/paper_list/industry.md)
  
We have also summarized some conventional methods for different tasks.
* [Conventional Methods Camera 3D Object Detection](docs/paper_list/camera_detection.md)
* [Conventional Methods LiDAR Detection](docs/paper_list/lidar_detection.md)
* [Conventional Methods LiDAR Segmentation](docs/paper_list/lidar_segmentation.md)
* [Conventional Methods Sensor Fusion](docs/paper_list/sensor_fusion.md)

## BEV Toolbox
The BEV toolbox provides useful recipes for BEV camera-based 3D object detection, including solid data augmentation strategies, efficient BEV encoder design, loss function family, useful test-time augmentation, ensemble policy, and so on. Please refer to [bev_toolbox/README.md](bev_toolbox/README.md) for more details.

## BEV Knowledge Distillation Algorithms
The BEV algorithm family includes follow-up works of BEVFormer in different aspects, ranging from plug-and-play tricks to pre-training distillation. All paper summary is under [nuscenes_playground](nuScenes_playground) along with official implementation, check it out!
* **VCD (NeurIPS 2023)** <br> [Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection](https://arxiv.org/abs/2310.15670). More details can be found in [nuScenes_playground/VCD](./nuScenes_playground/VCD/README.md).
* **GAPretrain (arXiv)** <br> [Geometric-aware Pretraining for Vision-centric 3D Object Detection](https://arxiv.org/abs/2304.03105). More details can be found in [nuScenes_playground/GAPretrain.md](./nuScenes_playground/GAPretrain.md).
* **FocalDistiller (CVPR 2023)** <br> [Distilling Focal Knowledge from Imperfect Expert for 3D object Detection](https://openaccess.thecvf.com/content/CVPR2023/html/Zeng_Distilling_Focal_Knowledge_From_Imperfect_Expert_for_3D_Object_Detection_CVPR_2023_paper.html). More details can be found in [nuScenes_playground/FocalDistiller](nuScenes_playground/FocalDistiller/README.md).

## License and Citation
This project is released under the [Apache 2.0 license](LICENSE).

If you find this project useful in your research, please consider cite:

```BibTeX
@article{li2022bevsurvey,
  author={Li, Hongyang and Sima, Chonghao and Dai, Jifeng and Wang, Wenhai and Lu, Lewei and Wang, Huijie and Zeng, Jia and Li, Zhiqi and Yang, Jiazhi and Deng, Hanming and Tian, Hao and Xie, Enze and Xie, Jiangwei and Chen, Li and Li, Tianyu and Li, Yang and Gao, Yulu and Jia, Xiaosong and Liu, Si and Shi, Jianping and Lin, Dahua and Qiao, Yu},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Delving Into the Devils of Bird's-Eye-View Perception: A Review, Evaluation and Recipe}, 
  year={2023},
  volume={},
  number={},
  pages={1-20},
  doi={10.1109/TPAMI.2023.3333838}
}
```
```BibTeX
@misc{bevtoolbox2022,
  title={{BEVPerceptionx-Survey-Recipe} toolbox for general BEV perception},
  author={BEV-Toolbox Contributors},
  howpublished={\url{https://github.com/OpenDriveLab/Birds-eye-view-Perception}},
  year={2022}
}
```