nopa

KO | EN

gitlite — search

#framework #postgresql #ui #computer-vision #graphql #pdf #ios #vue #csharp #hacktoberfest

(★ 9)

[ECCV 2026] NoPA: Non-Parametric Online 3D Scene Graph Generation

파일 탐색기

ZIP 다운로드

- - __init__.py
  - open_image.py
  - threerscan.py
  - visual_genome.py
- - __init__.py
  - pytorch_misc.py
- - architecture.png
- - __init__.py
  - deformable_detr.py
  - egtr.py
  - load_custom.py
  - rtdetr_egtr.py
  - tensorrt_inference.py
  - transform.py
  - util.py
- - __init__.py
  - box_ops.py
  - misc.py
- .gitignore
- evaluate_egtr.py
- evaluate_rtdetr_egtr.py
- LICENSE
- NOTICE
- pretrain_detr.py
- README.md
- requirements.txt
- train_egtr.py
- train_rtdetr_egtr.py
- evaluate.py
- global_sg_filter.py
- keyframe_selection.py
- main.py
- particle_sg.py
- scan_transforms.py
- sg_loader.py
- sg_prediction.py
- utils.py
- - boxes2coco.py
  - extract_and_preprocess_3RScan.py
  - relationship2scannet.py
- - 3dssg_to_scannet.json
  - rio_renderer.patch
  - train_scans.txt
  - validation_scans.txt
- - export_onnx_trt.py
- prepare_datasets.sh
.gitignore
LICENSE
README.md
requirements.txt

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

</div>

## Overview
NoPA is non-parametric-based online 3D semantic scene graph (SSG) generator that lifts 2D scene graphs predicted from RGB-D images into a global 3D scene graph. Objects are represented by non-parametric particle set distributions, and cross-view association / merge decisions use a Hellinger-distance pre-filter followed by an MMD refinement. Additional relationship propagation retains predicate observations that would otherwise be lost when tracks are split by strict association thresholds.

## Table of Contents
- [Installation](#installation)
- [Prepare Dataset](#prepare-dataset)
  - [Download 3RScan dataset](#1-download-3rscan-dataset)
  - [Extract and preprocess 3RScan dataset](#2-extract-and-preprocess-3rscan-dataset)
  - [Prepare datasets for object detection, 2D scene graph generation, and 3D scene graph generation](#3-prepare-datasets-for-object-detection-2d-scene-graph-generation-and-3d-scene-graph-generation)
  - [Download ReplicaSSG dataset](#4-download-replicassg-dataset)
- [Download Pretrained RT-DETR-EGTR Weights](#download-pretrained-rt-detr-egtr-weights)
- [Run NoPA](#run-nopa)
- [Evaluate NoPA](#evaluate-nopa)
- [Citation](#citation)
- [References](#references)

## Installation
Tested with Python 3.9 and CUDA 12.1 on Ubuntu 22.04.

### Prerequisites
- libvips-dev
- A working TensorRT installation matching the CUDA version above (required by the RT-DETR-EGTR inference path).

### Create the Conda environment
```bash
conda create -n nopa python=3.9 -y
conda activate nopa
```

### Install Dependencies
```bash
git clone <your-nopa-remote>.git NoPA
cd NoPA
pip install torch==2.3.0 torchvision==0.18.0 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt -f https://data.pyg.org/whl/torch-2.3.0+cu121.html
```

## Prepare Dataset
NoPA supports the 3RScan (via 3DSSG_subset with ScanNet-20 classes) and Replica (via ReplicaSSG with Visual-Genome classes) datasets.

#### 1. Download 3RScan dataset
Agree to the terms of use and get the download script from [here](https://forms.gle/NvL5dvB4tSFrHfQH6) and save it as `3RScan.py`.
```bash
python 3RScan.py -o Datasets/3RScan/data
wget "http://campar.in.tum.de/public_datasets/3RScan/3RScan.json" -P Datasets/3RScan/data
wget "http://campar.in.tum.de/public_datasets/3DSSG/3DSSG/objects.json" -P Datasets/3RScan/data
wget "http://campar.in.tum.de/public_datasets/3DSSG/3DSSG/relationships.json" -P Datasets/3RScan/data
```

#### 2. Extract and preprocess 3RScan dataset
Build the `rio_renderer` per the [3RScan repository](https://github.com/WaldJohannaU/3RScan/tree/master/c%2B%2B). Once it is built, render depth and label maps for every scene:

```bash
python3 Scripts/dataset/extract_and_preprocess_3RScan.py \
  --path ./Datasets/3RScan/ \
  --rio_renderer_path <path-to-rio_renderer/build>
```

#### 3. Prepare datasets for object detection, 2D scene graph generation, and 3D scene graph generation
```bash
cd Scripts
bash prepare_datasets.sh
cd ..
```
This runs:
- `Scripts/dataset/relationship2scannet.py` — filters `3DSSG_subset/relationships.json` down to the 20-class ScanNet subset used by NoPA.
- `Scripts/dataset/boxes2coco.py` — converts per-frame bounding boxes into COCO-format `train/val/test.json` files.

#### 4. Download ReplicaSSG dataset
Follow the [ReplicaSSG](https://github.com/Howardkhh/ReplicaSSG) instructions to obtain the dataset, then move it into `./Datasets`:
```bash
mv <path-to>/ReplicaSSG/Replica ./Datasets
```

Convert the Replica 2D annotations to COCO format so NoPA can consume them:
```bash
python Scripts/dataset/boxes2coco.py --path ./Datasets/Replica --label_categories replica
```

## Download Pretrained RT-DETR-EGTR Weights
Download the pretrained scene-graph generator (RT-DETR-EGTR) weights (same weights as FROSS):
- [3RScan](https://drive.google.com/file/d/1k7PLsY0CqbZbBHeKU8yA2Eof8wFh4Hap/view?usp=sharing)
- [ReplicaSSG (VG)](https://drive.google.com/file/d/1glMkDC1UPQbd8JfjQa6VzNQRwMDAnOsI/view?usp=sharing)

Unzip into `weights/RT-DETR-EGTR_pretrained/`:
```bash
mkdir -p weights/RT-DETR-EGTR_pretrained
cd weights/RT-DETR-EGTR_pretrained
unzip 3RScan20.zip
unzip VG.zip
cd ../..
```

Export to TensorRT engines (NoPA's inference path expects `rt-detr.engine` and `egtr-head.engine` inside `--artifact_path`):
```bash
# 3RScan
PYTHONPATH=. python Scripts/tools/export_onnx_trt.py \
  --artifact_path weights/RT-DETR-EGTR_pretrained/3RScan20/egtr__RT-DETR__3RScan20__last.pth/batch__6__epochs__50_25__lr__2e-07_2e-06_0.0002__finetune/version_0

# ReplicaSSG
PYTHONPATH=. python Scripts/tools/export_onnx_trt.py \
  --artifact_path weights/RT-DETR-EGTR_pretrained/VG/egtr__RT-DETR__VG__last.pth/batch__6__epochs__50_25__lr__2e-07_2e-06_2e-05__finetune/version_0
```

## Run NoPA
<details>
<summary><code>main.py</code> selected parameters</summary>

- `--artifact_path`: Directory containing the exported `rt-detr.engine` / `egtr-head.engine` and `config.json`.
- `--dataset_path`: Path to the Replica or 3RScan root.
- `--label_categories`: `scannet` (3RScan) or `replica`.
- `--not_preload`: Disable the frames-in-RAM cache (recommended when the dataset is large).
- `--use_hellinger_prefilter` / `--hellinger_margin`: Enable the Hellinger pre-filter that skips MMD for definite associate/spawn cases.
- `--mmd_threshold`: MMD gating threshold for particle-cloud associations and merges.
- `--rel_topk`: Number of top predicates the RT-DETR-EGTR head keeps per frame.
- `--n_particles`: Particle count per object track.
- `--use_soft_relations` / `--affinity_threshold`: Enable soft relationship registration and affinity resolution during finalization.
</details>

Predicted scene graphs (per scan) are written under `output/<label_categories>/`.

```bash
cd Merging

# 3RScan dataset (reference)
python main.py \
  --artifact_path ../weights/RT-DETR-EGTR_pretrained/3RScan20/egtr__RT-DETR__3RScan20__last.pth/batch__6__epochs__50_25__lr__2e-07_2e-06_0.0002__finetune/version_0/\ 
  --dataset_path ../Datasets/3RScan \
  --not_preload \
  --use_hellinger_prefilter --hellinger_margin 0.05 \
  --mmd_threshold 0.7 --rel_topk 20 --n_particles 256 \
  --use_soft_relations --affinity_threshold 0.25

# ReplicaSSG dataset (this is the reference command)
python main.py \
  --artifact_path ../weights/RT-DETR-EGTR_pretrained/VG/egtr__RT-DETR__VG__last.pth/batch__6__epochs__50_25__lr__2e-07_2e-06_2e-05__finetune/version_0/ \
  --dataset_path ../Datasets/Replica \
  --label_categories replica \
  --not_preload \
  --use_hellinger_prefilter --hellinger_margin 0.05 \
  --mmd_threshold 0.75 --rel_topk 20 --n_particles 256 \
  --use_soft_relations --affinity_threshold 0.25
```

The output pickle for the Replica command above will be at:
```
output/replica/predictions_filter_obj0.7_rel20_hell0.85_kfnone_test_gtpose.pkl
```

## Evaluate NoPA
```bash
cd Merging

# 3RScan dataset — MUST pass --use_aligned_ply so the GT mesh and predictions live in the same
# coordinate frame (labels.instances.align.annotated.v2.ply). Omitting it drops Recall@1 by ~40%.
python evaluate.py \
  --dataset_path ../Datasets/3RScan/ \
  --prediction_path output/scannet/predictions_filter_obj0.7_rel10_hell0.85_kfnone_test_gtpose.pkl \
  --use_aligned_ply

# ReplicaSSG dataset
python evaluate.py \
  --dataset_path ../Datasets/Replica/ \
  --label_categories replica \
  --prediction_path output/replica/predictions_filter_obj0.7_rel20_hell0.85_kfnone_test_gtpose.pkl
```
`evaluate.py` prints per-class and aggregated Recall@1/5/10 for objects, predicates, and relationships.

## Citation
If you use NoPA in academic work, please cite the paper this repository builds on:
```
@misc{yeo2026nopa,
      title={{NoPA}: Non-Parametric Online 3D Scene Graph Generation}, 
      author={Qi Xun Yeo and Seungjun Lee and Yan Li and Gim Hee Lee},
      year={2026},
      eprint={2607.00529},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00529}, 
}
```

## References
This repository would not exist without building on pioneering code from the following great works:
- [FROSS](https://github.com/Howardkhh/FROSS)
- [RT-DETR](https://github.com/lyuwenyu/RT-DETR)
- [EGTR](https://github.com/naver-ai/egtr)
- [3RScan](https://github.com/WaldJohannaU/3RScan)
- [3DSSG](https://3dssg.github.io/)
- [Replica](https://github.com/facebookresearch/Replica-Dataset)
- [ReplicaSSG](https://github.com/Howardkhh/ReplicaSSG)
- [Visual Genome](https://homes.cs.washington.edu/~ranjay/visualgenome/index.html)
- [JointSSG/MonoSSG, Wu et al.](https://github.com/ShunChengWu/3DSSG)

nopa

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README