R3D2 - GitLite

KO | EN

gitlite — search

#electron #agent #artificial-intelligence #deep-learning #javascript #mysql #chatgpt #open-source #pytorch #ios

R3D2

(★ 80)

Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation

파일 탐색기

ZIP 다운로드

- method_diagram.png
- - config.py
  - train_config.py
- - r3d3.py
- - base.py
  - r3d2.py
- - perceptual_loss.py
- - eval.py
  - measure_fps.py
  - run_on_video.py
  - train.py
  - validate.py
- - pix2pix_trainer.py
- - config_utils.py
  - device_utils.py
  - export_pipeline.py
  - get_pipeline.py
  - image_utils.py
  - sfast.py
.gitignore
LICENSE
pyproject.toml
README.md
uv.lock

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`ffmpeg`

설명 보기 ▼

비디오 변환 도구.
관련 항목: `gst-launch-1.0`.
더 많은 정보: <https://ffmpeg.org/ffmpeg.html#Options>.

ffmpeg -i {{path/to/video.mp4}} -vn {{path/to/sound.mp3}}

비디오에서 사운드를 추출하여 MP3로 저장:

ffmpeg -i {{path/to/input_audio.flac}} -ar 44100 -sample_fmt s16 {{path/to/output_audio.wav}}

FLAC 파일을 Red Book CD 형식 (44100kHz, 16bit)으로 트랜스코딩:

ffmpeg -i {{path/to/video.mp4}} {{[-vf|-filter:v]}} 'scale=-1:1000' -r 15 {{path/to/output.gif}}

비디오를 GIF로 저장하고, 높이를 1000px로 조정하고 프레임 속도를 15로 설정:

🔍

`uv run`

설명 보기 ▼

Run a command or script in the project environment.

uv run {{path/to/script.py}}

Run a Python script:

uv run {{[-m|--module]}} {{module_name}}

Run a Python module:

uv run {{[-w|--with]}} {{package}} {{command}}

Run a command with additional packages installed temporarily:

🔍

`uv sync`

설명 보기 ▼

Update the project's environment to match the lockfile.

uv sync

Sync the project environment with the lockfile:

uv sync --all-extras

Sync and include all optional dependencies:

uv sync --extra {{extra_name}}

Sync with specific optional dependencies:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">

<h1>R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation</h1>
<p><em>CVPR Workshop on Autonomous Driving 2026</em></p>

</div>

<p align="center">
    
    <a href="https://research.zenseact.com/publications/R3D2/"><img src="https://img.shields.io/badge/Project-Page-ffa"/></a>
    
    <a href="https://www.arxiv.org/abs/2506.07826">
        <img src='https://img.shields.io/badge/arXiv-Paper-aff'>
    </a>
</p>

<p align="center">
  <img src="assets/method_diagram.png" alt="R3D2 method diagram" width="80%"/>
</p>

[Project page](https://research.zenseact.com/publications/R3D2/)  ·
[Paper](https://www.arxiv.org/abs/2506.07826)  ·
[Setup](#setup)  ·
[Usage](#usage)  ·
[Citation](#citation)

</div>

## About

This is the official code release for *R3D2: Realistic 3D Asset Insertion via Diffusion for
Autonomous Driving Simulation*. R3D2 is a lightweight, one-step diffusion model that inserts complete
3D assets into existing neural-rendered scenes by generating plausible rendering effects — shadows and
consistent lighting — in real time.

## Abstract

Validating autonomous driving (AD) systems requires diverse and safety-critical testing, making
photorealistic virtual environments essential. Traditional simulation platforms, while controllable,
are resource-intensive to scale and often suffer from a domain gap with real-world data. In contrast,
neural reconstruction methods like 3D Gaussian Splatting (3DGS) offer a scalable solution for creating
photorealistic digital twins of real-world driving scenes. However, they struggle with dynamic object
manipulation and reusability as their per-scene optimization-based methodology tends to result in
incomplete object models with integrated illumination effects. This paper introduces R3D2, a
lightweight, one-step diffusion model designed to overcome these limitations and enable realistic
insertion of complete 3D assets into existing scenes by generating plausible rendering effects—such as
shadows and consistent lighting—in real time. This is achieved by training R3D2 on a novel dataset:
3DGS object assets are generated from in-the-wild AD data using an image-conditioned 3D generative
model, and then synthetically placed into neural rendering-based virtual environments, allowing R3D2 to
learn realistic integration. Quantitative and qualitative evaluations demonstrate that R3D2
significantly enhances the realism of inserted assets, enabling use-cases like text-to-3D asset
insertion and cross-scene/dataset object transfer, allowing for true scalability in AD validation.

## Setup

Dependencies are managed with `[uv](https://github.com/astral-sh/uv)` and locked in `uv.lock`.

```bash
uv sync
```

This installs the `r3d2` package in editable mode and all dependencies into a local `.venv`. Base
weights (`stabilityai/sd-turbo` and, for the tiny-VAE variants, `madebyollin/taesd`) are downloaded
from the Hugging Face Hub on first run.

## Dataset

Point a config at a directory containing a `metadata.json` with this structure:

```json
{
  "root": "/absolute/path/to/render/root",
  "train": [
    {"seq": "scene-0001", "gt": "rel/path/gt/FRONT_00000001.jpg", "wo_shadow": "rel/path/pred/FRONT_00000001.jpg"}
  ],
  "val": [
    {"seq": "scene-0002", "gt": "...", "wo_shadow": "..."}
  ]
}
```

- `gt` is the ground-truth image; `wo_shadow` is the rendered prediction used as the model input.
- `gt`/`wo_shadow` paths are relative to the top-level `root`.
- By default only `FRONT` camera frames are used (`cams_to_use=("FRONT",)`); set `--data.cams-to-use`
to change this, or pass `None` to use every frame. Filenames are expected to be `<CAMERA>_<index>`.

## Usage

Scripts are configured with `[tyro](https://github.com/brentyi/tyro)`. Three predefined
configurations are available:

| Config      | VAE     | Resolution | Notes                                      |
| ----------- | ------- | ---------- | ------------------------------------------ |
| `R3D2`      | tiny    | 1080×1920  | Main model from the paper; fast inference. |
| `R3D2-big`  | regular | 1080×1920  | Higher quality, slower.                    |
| `R3D2-fast` | tiny    | 512×512    | Quick experiments / lower compute.         |

### Training

```bash
uv run r3d2/scripts/train.py R3D2 --data.data-path /path/to/dataset
```

Any config field can be overridden on the command line, e.g.:

```bash
uv run r3d2/scripts/train.py R3D2-big --data.data-path /path/to/dataset --trainer.train-cfg.batch-size 2
```

Training logs to [Weights & Biases](https://wandb.ai). Set `--trainer.train-cfg.wandb-entity` and
`--trainer.train-cfg.project-name`, or run `wandb offline` to disable uploads. Checkpoints and the
exported pipeline are written under `outputs/`.

### Validation

```bash
uv run r3d2/scripts/validate.py R3D2 --data.data-path /path/to/dataset \
    --trainer.train-cfg.resume-from-checkpoint /path/to/checkpoint
```

### Inference

Single image (on an exported pipeline directory):

```bash
uv run r3d2/scripts/eval.py --model-path outputs/<run>/model --image-path input.jpg --output-path out.png
```

Video (requires `ffmpeg` on the system path):

```bash
uv run r3d2/scripts/run_on_video.py --model-path outputs/<run>/model --video-path in.mp4 --output-path out.mp4
```

### FPS benchmark

`measure_fps.py` benchmarks inference throughput (real-time speed is a core feature of R3D2):

```bash
uv run r3d2/scripts/measure_fps.py --model-path outputs/<run>/model
# or benchmark the sd-turbo baseline without a trained model:
uv run r3d2/scripts/measure_fps.py --no-r3d2 --tiny-vae
```

Optional acceleration via `[stable-fast](https://github.com/chengzeyi/stable-fast)`
(`--compile-type sfast`) and Triton.

## Citation

If you find this work useful, please consider citing:

```bibtex
@inproceedings{ljungbergh2025r3d2,
  title        = {R3D2: Realistic 3D Asset Insertion via Diffusion for Autonomous Driving Simulation},
  author       = {Ljungbergh, William and Taveira, Bernardo and Zheng, Wenzhao and Tonderski, Adam and Peng, Chensheng and Kahl, Fredrik and Petersson, Christoffer and Felsberg, Michael and Keutzer, Kurt and Tomizuka, Masayoshi and Zhan, Wei},
  booktitle    = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
  year         = {2026}
}
```

## License

Apache License 2.0. See [LICENSE](LICENSE).