ObAct

KO | EN

gitlite — search

#cli #chatbot #developer-tools #anthropic #command-line #ui #rust #swift #http #docker

(★ 9)

A lightweight implementation of the view optimization method from "Observer-Actor: Active Vision Imitation Learning with Sparse-View Gaussian Splatting" [ICRA 2026].

File Explorer

ZIP Download

- - config.yaml
  - reference_mask.png
  - reference_rgb.png
- - instantsplat_bridge.py
  - reconstruction.py
  - refinement.py
  - runtime.py
  - scoring.py
- __init__.py
- backend.py
- cli.py
- config.py
- dataset.py
- geometry.py
- matching.py
- pipeline.py
- sampling.py
- types.py
- run_example.py
- visualize_reconstruction.py
- - fake_instantsplat_backend.py
- test_backend.py
- test_bridge.py
- test_dataset.py
- test_failures.py
- test_geometry.py
- test_integration.py
- test_matching.py
- test_pipeline.py
- test_refinement.py
- test_sampling.py
.gitignore
pyproject.toml
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Minimal, hardware-agnostic repository of the offline observer stage from ObAct.

Given three recorded RGB-D observations, camera intrinsics, object masks, and camera-to-world poses, this package samples candidate next views, reconstructs with an external `InstantSplat` checkout, scores rendered candidates against a reference image, and returns the selected next camera pose in the original world frame.

`InstantSplat` receives only the three RGB images. Depth, masks, intrinsics, and poses are used by ObAct for candidate sampling, scoring, and frame alignment.

## Scope

Included:

- Offline observer pipeline: three sparse RGB-D views to a next-best-view pose.
- A tiny recorded mug example with segmentation masks.
- Pure Python dataset loading, pose alignment, candidate sampling, and result packaging.
- A thin bridge to the original external `InstantSplat` repo.
- Optional local pose refinement through the original InstantSplat Gaussian renderer.

Not included:

- ROS, MoveIt, URDFs, robot drivers, live camera topics, or policy execution.
- `InstantSplat` source, checkpoints, or model weights.
- Segmentation models. Masks are precomputed inputs in this release.

## Repository Layout

- `obact/`: reusable package.
- `examples/mug_demo/inputs/`: three-view example input bundle.
- `examples/mug_demo/config.yaml`: public example config.
- `examples/mug_demo/outputs/mug_demo/`: curated example outputs from a successful run.
- `scripts/run_example.py`: one-command entrypoint.
- `scripts/visualize_reconstruction.py`: optional point-cloud visualization.
- `tests/`: lightweight package tests.

## Installation

Install the package:

```bash
cd obact_public
python3 -m pip install -e .
```

For the real InstantSplat-backed example, first set up the original `InstantSplat` environment, then install ObAct's optional matcher and refinement dependencies into that environment:

```bash
python -m pip install -e .[matching,refinement]
```

The `matching` extra requests `romatch`, `torch`, and `torchvision`; keep the Torch build compatible with the InstantSplat CUDA extensions in your environment. ObAct uses [RoMa](https://github.com/Parskatt/RoMa) dense matching for view ranking, and RoMa downloads model weights on first use. If `romatch` is unavailable from PyPI in your environment, install it directly:

```bash
python -m pip install git+https://github.com/Parskatt/RoMa.git
```

Keep `InstantSplat` outside this repo. The example config expects a local checkout path and calls ObAct's bridge into that checkout.

## Run The Example

Edit `examples/mug_demo/config.yaml` and set:

```json
"instantsplat_root": "/absolute/path/to/InstantSplat"
```

Then run:

```bash
python scripts/run_example.py --config examples/mug_demo/config.yaml
```

For machine-specific paths, copy the config to `examples/mug_demo/config.local.json`. That filename is gitignored so private paths do not leak into the public repo.

## Input Contract

Each sequence contains exactly three observations:

- `rgb_001.png`, `rgb_002.png`, `rgb_003.png`
- `depth_001.npy`, `depth_002.npy`, `depth_003.npy`
- `mask_001.png`, `mask_002.png`, `mask_003.png`
- `pose_001.npy`, `pose_002.npy`, `pose_003.npy`
- `intrinsics.npy`

Reference files live next to the sequence:

- `reference_rgb.png`
- `reference_mask.png`

All poses are `4x4` camera-to-world transforms. For custom datasets, override `rgb_files`, `depth_files`, `mask_files`, `pose_files`, or `reconstruction_rgb_files` in the config.

## Outputs

The configured output directory contains:

- `best_view_pose.npy`: selected next view in the input world frame.
- `render_preview.png`: side-by-side selection preview.

When `backend.refinement.enabled` is true, the pipeline also writes:

- `refined_view_pose.npy`
- `refinement_loss.png`
- `refinement.mp4`

The default refinement starts from the best coarse candidate only (`top_k: 1`) and locally optimizes the camera pose with a bounded SE(3) update through the original InstantSplat Gaussian renderer. It is a local refinement step, not a global optimality guarantee.

## Backend Notes

ObAct does not vendor `InstantSplat`. Instead it:

1. Creates a temporary workspace with the three reconstruction RGB images.
2. Runs external reconstruction through the configured backend command.
3. Aligns InstantSplat poses back to the input world frame.
4. Renders and scores candidate views with RoMa dense matching inside the reference mask.
5. Optionally refines the selected pose with differentiable rendering.

The public pipeline is intentionally hardware-agnostic at the interface level: any recorded RGB-D source can be used as long as it provides the files above.

## Visualization

After running the example, you can visualize the fused RGB-D observations and selected camera frustums:

```bash
python -m pip install -e .[viz]
python scripts/visualize_reconstruction.py
```

This writes `fused_reconstruction.ply` to the example output directory.

## Citation

If you find this repository useful for your project, please consider citing us!

```bibtex
@article{wang2025observer,
  title={Observer Actor: Active Vision Imitation Learning with Sparse View Gaussian Splatting},
  author={Wang, Yilong and Qian, Cheng and Fan, Ruomeng and Johns, Edward},
  journal={arXiv preprint arXiv:2511.18140},
  year={2025}
}
```

ObAct

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

ObAct

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README