veritas

KO | EN

gitlite — search

#command-line #html #swift #artificial-intelligence #rust #privacy #gpt #monitoring #cli #svg

(★ 9)

[RSS26] Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement

파일 탐색기

ZIP 다운로드

- teaser_veritas.png
- - bridge.yaml
- - bridge.yaml
- bridge_statistics.json
- - oxe.py
  - run_paligemma.py
  - sampling.py
  - simpler.py
- run.py
- set_path.sh
- try_checkpoint_in_simpler.py
- - dataset.py
  - eval.py
  - eval_verifier_utils.py
  - model_averaging.py
  - object_tracker.py
  - tracking_server.py
  - tracking_server_core.py
  - train.py
  - verified_data_collector.py
- - dataset.py
  - dataset_torch.py
  - obs_transforms.py
  - traj_transforms.py
- - kv_cache.py
  - lora.py
  - utils.py
- - decorator.py
  - geometry.py
  - metric.py
  - monitor.py
  - optim.py
  - spec.py
.gitignore
.gitmodules
.pre-commit-config.yaml
eval_with_veritas_verifier.sh
Grounded-SAM-2
LICENSE
pyproject.toml
README.md
SimplerEnv
uv.lock

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

---

**[🌐 Website](https://veritas-improvement.github.io/)**

*Visual Verification Enables Inference-time Steering and Autonomous Policy Improvement* is a generator–verifier framework for generalist robot policies. A pre-trained policy proposes candidate action chunks; a gradient-free visual verifier scores them in pixel space against a Gemini-emitted waypoint plan and steers execution to the best one at inference time. The same verified rollouts can be recycled as new training data, enabling continual policy self-improvement without human interventions.

## Installation

### Clone the repository (recursively)

```bash
git clone --recursive git@github.com:robo-alex/veritas.git
cd veritas
```

If you cloned without `--recursive`:

```bash
git submodule update --init --recursive
```

### Setup environment with uv

See [installation instructions](https://docs.astral.sh/uv/getting-started/installation/).

We use CUDA 13 by default. If you have an another version of CUDA installed, please change the torch / torchvision versions and index URL to be compatible in the [pyproject.toml](pyproject.toml).

```bash
uv sync
```

### Download PaliGemma weights

```bash
export HF_HUB_CACHE=$PWD/transformers_cache
mkdir -p $HF_HUB_CACHE
git clone https://huggingface.co/google/paligemma-3b-pt-224 $HF_HUB_CACHE/paligemma-3b-pt-224
```

### Download a SAM2 + Grounding-DINO checkpoint

The Veritas verifier uses SAM2 hiera_large + Grounding-DINO base for object localization. From the repo root:

```bash
bash Grounded-SAM-2/checkpoints/download_ckpts.sh
```

### Download the open-pi-zero policy checkpoint

Pull the Bridge policy from HuggingFace into `models/`.

```bash
mkdir -p models
uvx hf download allenzren/open-pi-zero \
  bridge_uniform_step19296_2024-12-26_22-31_42.pt \
  --local-dir models
```

## Getting Started

### Minimal code example

A raw rollout with no verifier — useful to confirm the env, checkpoint, and venv are wired up:

```python
import hydra
from omegaconf import OmegaConf
from src.agent.eval import EvalAgent

cfg = OmegaConf.load("config/eval/bridge.yaml")
cfg.env.task = "widowx_put_eggplant_in_basket"
cfg.checkpoint_path = "models/bridge_uniform_step19296_2024-12-26_22-31_42.pt"
cfg.use_veritas_verifier = False
cfg.verifier_samples = 1
cfg.log_dir = "logs/raw/eval_widowx_put_eggplant_in_basket"

agent = EvalAgent(cfg)
agent.run()
```

### Run the Veritas verifier on all WidowX tasks

```bash
export GEMINI_API_KEY=...
bash eval_with_veritas_verifier.sh
```

Each task runs 240 episodes (24 variations × 10 trials). Per-episode `.mp4` videos land under `logs/verifier_eval/eval_<task>/videos/`.

## Evaluation Tasks

| Task | Prompt |
|:---|:---|
| `widowx_put_eggplant_in_basket` | put eggplant into yellow basket |
| `widowx_stack_cube` | stack the green cube on top of the yellow cube |
| `widowx_carrot_on_plate` | put carrot on plate |
| `widowx_spoon_on_towel` | put the spoon on the towel |

## How it works

1. **Plan.** At episode init, Gemini emits pixel-space waypoints in the policy camera's image: `(u, v)` plus per-waypoint `tol_px` and `min_hold`.
2. **Sample.** Each chunk decision draws 5 candidate action chunks from the policy (one batched forward pass through PaliGemma + action expert × 10 flow-matching steps).
3. **Score.** Each candidate's predicted EE endpoint is projected to pixels and scored against the upcoming K waypoints.
4. **Step.** Execute the best-scoring chunk, advance the active waypoint via hysteresis dwell, transit-skip, or stalled-skip.

## More details

This codebase has been tested on CUDA 13 with NVIDIA RTX PRO 6000 Blackwell and RTX 5090.

## Acknowledgements

The Pi0 model and PaliGemma setup are adapted from Allen's repo [open-pi-zero](https://github.com/allenzren/open-pi-zero) and [pytorch-paligemma](https://github.com/hkproj/pytorch-paligemma). Simulation environments are from [SimplerEnv](https://github.com/allenzren/SimplerEnv/tree/59ad9e1539042ed333fd8ebba1b0395f5662f0bd). Dataset loading is from [Octo](https://octo-models.github.io/) and [dlimp](https://github.com/kvablack/dlimp). Object detection is powered by [Grounded-SAM-2](https://github.com/IDEA-Research/Grounded-SAM-2).

veritas

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README