VIRENA

KO | EN

gitlite — search

#csharp #mysql #windows #http #react-native #css #self-hosted #web #claude #ios

(★ 9)

A minimal Vision-Language-Action model you can read: frozen CLIP + a tiny head on ManiSkill PickCube. LeRobot integration. Runs on a Mac, no GPU.

File Explorer

ZIP Download

- results.csv
- results.json
- RESULTS.md
- banner.png
- basecam_01_start.png
- basecam_02_approach.png
- basecam_03_grasp.png
- demo.gif
- so101_pred_vs_true.png
- so101_rollout.gif
- wristcam_01_start.png
- wristcam_02_approach.png
- wristcam_03_grasp.png
- - __init__.py
  - base_collector.py
  - maniskill_collector.py
- - __init__.py
  - camera_config.py
  - maniskill_wrapper.py
- - __init__.py
  - base_policy.py
  - random_policy.py
  - scripted_expert.py
- - __init__.py
  - episode_recorder.py
- __init__.py
- build_dataset.py
- collect.py
- export_lerobot.py
- inspect_data.py
- lerobot_schema.py
- __init__.py
- lerobot_reader.py
- virena_dataset.py
- __init__.py
- probe.py
- rollout.py
- tasks.py
- verdict.py
- ARCHITECTURE.md
- - __init__.py
  - head.py
- - __init__.py
  - encoder.py
- - __init__.py
  - encoder.py
  - pool.py
  - preprocess.py
- __init__.py
- vla.py
- vla_policy.py
- pickcube.pt
- ablation.py
- diagnose.py
- diagnose_sim.py
- eval_lerobot.py
- eval_sim.py
- predict.py
- record_demo.py
- test_dataloader.py
- test_lerobot_export.py
- test_lerobot_reader.py
- test_vision_encoder.py
- test_vla.py
- train.py
.gitignore
config.py
LICENSE
pyproject.toml
README.md
requirements-model.txt

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<!-- 
 A minimal Vision-Language-Action model you can actually read, 
 plus a case study in why VLAs fail and how to debug them.

Vision-language Imitation for Robots: Explainable, Nimble, Accessible. 
 From the Latin virēre, to flourish.
 -->

---

I built a small robot-learning model from scratch. It takes a camera image, an instruction, and the robot's joint state, and outputs motor commands to pick up a cube and place it at a target. It sat at 0% success for days. Bigger heads, more layers, different losses: none of it moved the number.

The fix was never the model. It was what the model could see. Twice, the thing that unblocked me was fixing the observation, not the architecture. Getting from 0% to a working policy turned out to be a debugging story rather than a modelling one, and that story is the most useful thing in this repo. (The exact number, 68% and later 93%, comes with a twist from the ablations further down.)

So VIRENA is really three things:

1. A working VLA in about 2,000 lines that trains on a laptop (frozen CLIP plus a tiny trained head, no GPU needed).
2. An ablation cookbook that measures what each trick is actually worth, with confidence intervals.
3. A failure-diagnosis toolkit that tells you whether your policy is failing because of perception, control, or data. That is the thing I had to work out by hand, now packaged so you don't have to.

It's small enough to read in an afternoon, it runs on a Mac without a GPU, and it's built to teach you how these systems actually work and how they fail. If you're learning how a VLA fits together, or you want a compact base to hack your own ideas onto, that's who it's for.

---

## The task

[`PickCube-v1`](https://github.com/haosulab/ManiSkill) in the ManiSkill3 simulator. A Panda arm has to pick up a red cube and place it at a random 3D goal, within 2.5 cm, then hold still. The scene is randomized every episode. The model only ever gets a 128x128 camera image, the arm's joint angles, and the goal coordinate.

<img src="assets/demo.gif" width="360" alt="VIRENA solving PickCube"/>
A trained rollout (external view). The policy itself only sees the small wrist image.

---

## The 30-second mental model

Only two small pieces get trained: the AttentionPool and the MLP head, about 950k parameters between them. Both CLIP towers stay frozen. That's why it trains in a couple of hours on an M2 and doesn't need a GPU. The full walk-through lives in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md), written for engineers who know PyTorch but not robotics. Every term is defined the first time it shows up.

---

## From 0% to a working policy

Every failure below was an observability problem. No model change fixed them. Changing what the model could observe did.

**Finding 1: the camera was blind at the worst possible moment.** The first setup used a fixed external camera. At the instant of grasping, the gripper itself covers the cube from that viewpoint. Watch the cube (the little red patch) disappear as the arm closes in:

<img src="assets/basecam_01_start.png" width="140" alt="base camera, start"/>
 <img src="assets/basecam_02_approach.png" width="140" alt="base camera, approach"/>
 <img src="assets/basecam_03_grasp.png" width="140" alt="base camera, grasp"/>

Before: fixed external camera. Cube clearly visible at the start, then swallowed by the gripper right when precision matters most. Success: 30%.

The symptom was easy to read once I looked for it: the hand-to-cube distance would improve, then flatten out at a hard floor around 3.3 cm, and it was the *same* floor for every architecture I tried. A floor that no model change can break is the fingerprint of a perception problem, not a learning one. You can't learn to close the last 3 cm to something you can't see.

The fix was a wrist-mounted camera (an "eye in hand") that keeps pointing at the cube through the grasp. Same cube, same grasp, but now the model can actually see what it's doing:

<img src="assets/wristcam_01_start.png" width="140" alt="wrist camera, start"/>
 <img src="assets/wristcam_02_approach.png" width="140" alt="wrist camera, approach"/>
 <img src="assets/wristcam_03_grasp.png" width="140" alt="wrist camera, grasp"/>

After: wrist camera. The cube stays centered and visible all the way into the grasp. Success: 68%. Same everything else, +38 points from the camera alone.

Same expert, same joint data, only the image source changed. The floor dropped to 1.5 cm and grasping started working. The ablation below puts a number on it: keeping everything else identical and swapping the wrist camera back for the external one drops success from 68% to 30%.

**Finding 2: the goal was literally invisible.** Even with grasping solved, placement was impossible. In `PickCube` the goal is a hidden marker at a random 3D point, and it isn't rendered in any camera. The model was being asked to move the cube to a location it had no way to perceive. The fix is goal-conditioning: feed the goal coordinate in as an explicit 3D vector. That coordinate is the task specification, not privileged internal state.

The takeaway, and the reason the two tools below exist: when a robot policy fails, first ask whether it can even see what it needs. Do that before you touch the model.

---

## The ablation cookbook: what each trick is worth

Every design choice is a single flag in [`config.py`](config.py), and [`scripts/ablation.py`](scripts/ablation.py) trains a fresh model for each variant, evaluates them all on the same 100 seeded scenes, and reports a Wilson 95% confidence interval. Turning off one component at a time turns "I think this helped" into a number:

| Ablation | Success | 95% CI | vs full |
|----------|--------:|:------:|:-------:|
| `no_chunking` (single-step, chunk=1) | **93%** | [86, 97] | **+25** |
| `full` (everything on) | 68% | [58, 76] | baseline |
| `pooled_vision` (global embedding, not patches) | 68% | [58, 76] | +0 |
| `no_history` (no proprio history) | 68% | [58, 76] | +0 |
| `no_proprio_norm` (raw proprio) | 61% | [51, 70] | -7 |
| `no_goal_norm` (raw goal) | 54% | [44, 63] | -14 |
| `base_camera` (external camera, occluded at grasp) | 30% | [22, 40] | -38 |
| `no_goal_cond` (drop the goal input) | 1% | [0, 5] | -67 |

Reading it honestly, including two results I did not expect:

- **Goal-conditioning is the backbone (-67).** Drop the goal vector and it collapses to 1%. That is Finding 2 quantified: with the goal invisible in the image, the model has nowhere to place the cube.
- **The wrist camera is worth +38.** `base_camera` is Finding 1 quantified: identical everything, occluded viewpoint, and success falls from 68% to 30%.
- **Normalizing inputs matters** (goal -14, proprio -7), the usual "keep your inputs on a sane scale" lesson.
- **Action chunking actually hurts here (+25 to remove it).** This is the surprise. I added chunking because ACT does, but for PickCube at this scale, plain single-step closed-loop prediction (chunk=1) scores 93%, well above the chunked `full` at 68%. Chunking commits to stale actions; re-deciding every step wins. It also reframes an earlier inference-only result where temporal ensembling looked worth +25 points: ensembling was partly repairing the damage chunking did to the chunked model, but plain single-step beats both.
- **Two things I built make no measurable difference here:** patch tokens vs the pooled CLIP embedding, and proprio history. Both land exactly at 68%. Good to know before I talk myself into believing they were essential.

---

## The failure-diagnosis toolkit: perception, control, or data?

`diagnose/` packages the debugging method into one command that returns a verdict of PERCEPTION, CONTROL, or DATA, with evidence. It combines a linear probe (can the frozen features even predict the actions?) with rollout signatures (distance floors, grasped-then-dropped, wandering).

```
$ python scripts/diagnose.py --checkpoint checkpoints/best.pt --seed 5

Perception probe   dim   probe R2
                    dx      0.754
                    dy      0.885     (features localise the cube fine)
                    dz      0.807
[rollout seed=5]  tcp_to_obj: min=0.028 (step 9) -> final=0.257
                  signatures: reached_then_left, grasped_then_lost

VERDICT: CONTROL  (confidence: high)
  - features encode the reach well, so perception is not the bottleneck
  - the target WAS grasped then lost, so it is clearly localizable
  -> Fix CONTROL: the failure is holding the grasp, not seeing it. If closed-loop
     is already on, the residual is fine-grained precision (edge-case data or resolution).
```

Point it at a policy and it tells you whether to fix the camera, the controller, or the data, with evidence instead of guesswork.

---

## Quick start

One conda environment does everything (data collection, training, evaluation, diagnosis). Installing `pinocchio` from conda-forge first is the trick that lets ManiSkill's IK and the training stack coexist; a plain `pip install` of both hits a numpy conflict. On macOS the scripts set `KMP_DUPLICATE_LIB_OK=TRUE` for you so you don't hit the duplicate-OpenMP abort.

```bash
conda create -n virena python=3.12 && conda activate virena
conda install -c conda-forge pinocchio
pip install "mani-skill==3.0.1" torch torchvision transformers Pillow tqdm imageio
```

### Try it without training

A pretrained single-step checkpoint (the 93% model) ships in the repo at `pretrained/pickcube.pt`, so after the install above you can run everything immediately, no data collection or training required:

```bash
python scripts/record_demo.py --checkpoint pretrained/pickcube.pt   # make the demo GIF
python scripts/eval_sim.py    --checkpoint pretrained/pickcube.pt --episodes 100
python scripts/diagnose.py    --checkpoint pretrained/pickcube.pt --seed 5
```

### Train it yourself

```bash
python -m data_collection.collect --num-episodes 1200 --output-dir data/raw --success-only
python -m data_collection.build_dataset --episodes-dir data/raw/episodes --output data/dataset.jsonl --success-only

# 2. Train (single-step is the default; writes checkpoints/best.pt)
python scripts/train.py --epochs 50

# 3. Evaluate and diagnose
python scripts/eval_sim.py --episodes 100
python scripts/diagnose.py --checkpoint checkpoints/best.pt --seed 5
```

> Optional: for fast model-only work (the perception probe, dataloader checks, training) you can use a lightweight `pip` venv with just `requirements-model.txt` and skip installing the simulator. It's a speed convenience, not a requirement.

---

## Project layout

```
config.py                 every design toggle in one dataclass (saved into each checkpoint)
data_collection/          scripted expert -> sim rollouts -> JSONL dataset
model/
  vision/                 frozen CLIP patch tokens plus a trainable AttentionPool
  language/               frozen CLIP text encoder
  action/                 the trainable MLP head (action chunking)
  vla.py                  fuses the four input streams
  vla_policy.py           inference: temporal ensembling plus the gripper latch
dataset/                  chunks, masks, episode-level split, normalization stats
diagnose/                 the perception/control/data toolkit (probe, rollout, verdict, tasks)
scripts/                  train, eval_sim, ablation, diagnose
docs/ARCHITECTURE.md      the full guided explanation
```

## LeRobot interoperability

VIRENA speaks the [LeRobot](https://github.com/huggingface/lerobot) dataset format both ways. It can export its own collected episodes so they load with the Hugging Face robotics stack, and it can train on real robot datasets pulled straight from the Hub. Both are thin adapters at the edge, so the core pipeline is untouched and `lerobot` stays an optional dependency.

One setup note: install `lerobot` in its own conda environment, not the simulation one. `lerobot` needs numpy 2.x, which clashes with ManiSkill's `pytorch-kinematics-ms` (numpy<2). Nothing here imports ManiSkill, so a separate env is all it takes.

```bash
conda create -n virena-lerobot python=3.12 && conda activate virena-lerobot
pip install "lerobot[dataset]>=0.4.0"   # the dataset extra pulls HuggingFace datasets
```

### Export VIRENA data to LeRobot

```bash
python -m data_collection.export_lerobot \
  --episodes-dir data/raw/episodes \
  --repo-id your-hf-user/virena-pickcube \
  --root data/lerobot/virena-pickcube \
  --success-only
  # add --push to upload to the Hub, or --as-images to skip MP4 encoding
```

The field mapping lives in one place ([`data_collection/lerobot_schema.py`](data_collection/lerobot_schema.py)): `state` becomes `observation.state`, `action` stays `action`, the wrist frame becomes `observation.images.wrist`, the instruction becomes the `task`, and the goal coordinate gets its own `observation.goal` feature (standard LeRobot datasets have no goal slot). A round-trip smoke test is in [`scripts/test_lerobot_export.py`](scripts/test_lerobot_export.py).

### Train VIRENA on real robot data

The more interesting direction is reading the format back in. [`dataset/lerobot_reader.py`](dataset/lerobot_reader.py) turns any LeRobot dataset into VIRENA training items (proprio history and action chunks come from LeRobot's `delta_timestamps`), and the training path adapts the model to whatever robot the dataset used. It auto-detects the proprio and action widths, drops goal-conditioning when there is no goal, and rescales out-of-range actions into the head's `[-1, 1]` range. So the same ~2,000 lines that solve PickCube also train on real teleoperated robot data with nothing changed but a flag.

To make the point, here is VIRENA trained on two deliberately different robots: a real single-arm SO-101 and a simulated bimanual ALOHA.

<img src="assets/so101_rollout.gif" width="280" alt="SO-101 pick-and-place rollout"/>

One episode from <a href="https://huggingface.co/datasets/lerobot/svla_so101_pickplace">lerobot/svla_so101_pickplace</a>, the real SO-101 arm VIRENA was trained on.

```bash
# real SO-101 pick-and-place (6-DoF single arm)
python scripts/train.py \
  --lerobot-repo-id lerobot/svla_so101_pickplace \
  --lerobot-image-key observation.images.up \
  --checkpoint-dir checkpoints/so101 --epochs 30

# offline eval on the held-out split, with a predicted-vs-truth plot
python scripts/eval_lerobot.py \
  --checkpoint checkpoints/so101/best.pt \
  --lerobot-repo-id lerobot/svla_so101_pickplace \
  --lerobot-image-key observation.images.up --plot
```

There is no simulator for these robots, so task success is not measurable. Instead the eval reports open-loop action-prediction error on held-out episodes: masked MSE over the action chunk, against a trivial predict-zeros baseline.

| Dataset | Robot | proprio / action | val MSE | predict-zeros |
|---|---|:---:|:---:|:---:|
| [`svla_so101_pickplace`](https://huggingface.co/datasets/lerobot/svla_so101_pickplace) | SO-101, real, single arm | 6 / 6 | **0.017** | 0.377 |
| [`aloha_sim_transfer_cube_human`](https://huggingface.co/datasets/lerobot/aloha_sim_transfer_cube_human) | ALOHA, sim, bimanual | 14 / 14 | **0.009** | 0.293 |

Both beat the baseline by more than 20x. The plot shows why the number is trustworthy: on a held-out SO-101 episode the predicted joint commands (orange) sit almost on top of the demonstrator's (blue).

The one honest weak spot is the gripper. On both robots the arm joints track cleanly, but the gripper channel (a5 on SO-101, and both grippers on ALOHA) is jittery, because it is a near-binary open/close signal that a plain MSE loss fits poorly. That is the obvious next thing to improve: a classification-style loss on the gripper dimension. A forward-pass smoke test for the reader is in [`scripts/test_lerobot_reader.py`](scripts/test_lerobot_reader.py).

Built on [ManiSkill3](https://github.com/haosulab/ManiSkill) and OpenAI [CLIP](https://github.com/openai/CLIP). Action chunking and temporal ensembling come from [ACT (Zhao et al. 2023)](https://tonyzhaozh.github.io/aloha/).

## License

[MIT](LICENSE). Do what you like with it; a link back is appreciated.

VIRENA

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README