Pipette

KO | EN

gitlite — search

#data-science #list #csharp #nodejs #github #http #openai #computer-vision #ui #machine-learning

(★ 9)

Preprint URL

File Explorer

ZIP Download

- README.md
- - pipette_overview.png
- - Close the centrifuge lid.gif
  - Close the spectrophotometer lid.gif
  - Open the centrifuge lid.gif
  - Open the water bath lid.gif
  - Pick up the test tube.gif
  - Place the centrifuge tube on the balance.gif
  - Place the petri dish in the incubator.gif
  - Place the pipette on the pipette stand.gif
  - Place_conical_flask_on_orbital_shaker.gif
  - Position the pipette over the petri dish.gif
  - Remove the centrifuge tube from the balance.gif
  - Remove the petri dish from the incubator.gif
- - Close the centrifuge lid.mp4
  - Close the spectrophotometer lid.mp4
  - Open the centrifuge lid.mp4
  - Open the water bath lid.mp4
  - Pick up the test tube.mp4
  - Place the centrifuge tube on the balance.mp4
  - Place the petri dish in the incubator.mp4
  - Place the pipette on the pipette stand.mp4
  - Place_conical_flask_on_orbital_shaker.mp4
  - Position the pipette over the petri dish.mp4
  - Remove the centrifuge tube from the balance.mp4
  - Remove the petri dish from the incubator.mp4
- - agent.py
  - hunyuan3d_client.py
  - llm_client.py
  - local_config.example.env
  - script_catalog.py
  - task_registration_rules.md
  - terminal_launcher.py
  - web_agent.py
- - create_physics_asset.py
- - inference_act.py
  - inference_pi0.py
  - inference_smolvla.py
  - policy_io.py
  - zmq_policy_client.py
- - delete_hdf5_demo.py
  - Enhance.py
  - Gamepad_collection.py
  - Generate_data.py
  - hdf5_to_lerobot.py
  - inspect_hdf5_dataset.py
  - Keyboard_collection.py
  - open_lid_evaluation.py
  - open_usd_environment.py
  - schema.py
  - task_eval_logging.py
  - task_registry.py
  - task_success_evaluator.py
  - temporal_resampling.py
  - virtual_button_lid_opener.py
- - inference_protocol.py
  - isaaclab_env_module.py
  - server_brain.py
README.md
README_zh-CN.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

### An Embodied Simulation Platform, Benchmark, and Data-Efficient Augmentation Framework for Wet-Lab Robotics

[English](README.md) | [简体中文](README_zh-CN.md)

[![Isaac Sim](https://img.shields.io/badge/Isaac%20Sim-5.1-76B900)](https://developer.nvidia.com/isaac-sim)
[![Isaac Lab](https://img.shields.io/badge/Isaac%20Lab-2.3-76B900)](https://isaac-sim.github.io/IsaacLab/)
[![LeRobot](https://img.shields.io/badge/LeRobot-ACT%20%7C%20SmolVLA%20%7C%20PI0-yellow)](https://github.com/huggingface/lerobot)
[![Python](https://img.shields.io/badge/Python-3.x-blue)](https://www.python.org/)

**Pipette** is an embodied-simulation toolkit for wet-lab robot learning. It provides a connected workflow for editing USD laboratory scenes, registering tasks, collecting teleoperation demonstrations, generating success-filtered simulation augmentations, converting HDF5 data to LeRobot, training policies, and running closed-loop evaluation.

## Platform Overview

## Repository Structure

```text
Pipette
|-- Asset/                      # Shared laboratory scenes, robots, and generated assets
|   |-- AI3D/                   # AI-generated 3D assets
|   |-- Asset/                  # Laboratory equipment and object assets
|   |-- Robot/                  # Robot assets
|   `-- Scene/                  # Laboratory scene assets
|-- docs/                       # Documentation images and demonstration media
|-- Scripts/                    # Data, training, inference, and evaluation scripts
|   |-- Agent/                  # Natural-language CLI and web orchestration
|   |-- Asset/                  # Assets used by the scripts
|   |-- Client/                 # Isaac Lab policy evaluation clients
|   |-- Data/                   # Collection, replay, augmentation, conversion, and evaluators
|   |-- Robot/                  # Robot registry and camera specifications
|   |-- Server/                 # Unified LeRobot ZMQ inference server
|   `-- run_lerobot_batch_train.py
|-- README.md
`-- README_zh-CN.md
```

> [!IMPORTANT]
> Before reproducing this project, download the complete asset package from [Pipette Asset](https://huggingface.co/datasets/huanbo1/pipette-asset). After downloading and extracting it, place `AI3D/`, `Asset/`, `Robot/`, and `Scene/` directly under the project-level `Asset/` directory (`Pipette/Asset/`, not `Pipette/Scripts/Asset/`) and preserve the directory structure shown above.

> [!TIP]
> If you prefer not to collect and convert data from scratch, download the provided datasets and place them in the corresponding locations:
>
> - [Pipette HDF5 Datasets](https://huggingface.co/datasets/huanbo1/pipette-hdf5-datasets): download and extract them to `Pipette/Scripts/datasets/` for inspection, replay, augmentation, and format conversion.
> - [Pipette LeRobot Datasets](https://huggingface.co/datasets/huanbo1/pipette-lerobot-datasets): download and extract them to your LeRobot dataset directory, such as `/path/to/lerobot/datasets/`, then point `--dataset-root` or `--dataset.repo_id` to the actual location when training.

## Highlights

- **Editable wet-lab assets:** standardized USD/USDZ assets with geometry, materials, collision bodies, physical properties, and task semantics.
- **12 wet-lab task presets:** sample handling, cultureware manipulation, equipment lid operation, precise placement, and object relocation.
- **Physically consistent augmentation:** trajectories are re-executed in Isaac Sim with lighting, camera, speed, and action perturbations.
- **Automatic success verification:** task-specific evaluators filter augmented episodes and record interpretable failure reasons.
- **LeRobot-ready data pipeline:** HDF5 demonstrations are converted to datasets containing synchronized top, main, and wrist RGB views, robot state, actions, and language instructions.
- **Unified policy evaluation:** ACT, SmolVLA, and PI0 share the same ZMQ policy interface and Isaac Lab evaluation environment.
- **Natural-language Agent:** a web interface orchestrates scene setup, task registration, data collection, augmentation, training, and evaluation.

## Benchmark Tasks

### Example Videos

<table>
 <tr>
 <td align="center"><img src="./docs/gif/Pick%20up%20the%20test%20tube.gif" alt="Pick up the test tube" width="100%"> Pick up the test tube</td>
 <td align="center"><img src="./docs/gif/Position%20the%20pipette%20over%20the%20petri%20dish.gif" alt="Position the pipette over the petri dish" width="100%"> Position the pipette over the petri dish</td>
 <td align="center"><img src="./docs/gif/Remove%20the%20petri%20dish%20from%20the%20incubator.gif" alt="Remove the petri dish from the incubator" width="100%"> Remove the petri dish from the incubator</td>
 <td align="center"><img src="./docs/gif/Place%20the%20petri%20dish%20in%20the%20incubator.gif" alt="Place the petri dish in the incubator" width="100%"> Place the petri dish in the incubator</td>
 </tr>
 <tr>
 <td align="center"><img src="./docs/gif/Close%20the%20centrifuge%20lid.gif" alt="Close the centrifuge lid" width="100%"> Close the centrifuge lid</td>
 <td align="center"><img src="./docs/gif/Open%20the%20centrifuge%20lid.gif" alt="Open the centrifuge lid" width="100%"> Open the centrifuge lid</td>
 <td align="center"><img src="./docs/gif/Open%20the%20water%20bath%20lid.gif" alt="Open the water bath lid" width="100%"> Open the water bath lid</td>
 <td align="center"><img src="./docs/gif/Close%20the%20spectrophotometer%20lid.gif" alt="Close the spectrophotometer lid" width="100%"> Close the spectrophotometer lid</td>
 </tr>
 <tr>
 <td align="center"><img src="./docs/gif/Place%20the%20centrifuge%20tube%20on%20the%20balance.gif" alt="Place the centrifuge tube on the balance" width="100%"> Place the centrifuge tube on the balance</td>
 <td align="center"><img src="./docs/gif/Remove%20the%20centrifuge%20tube%20from%20the%20balance.gif" alt="Remove the centrifuge tube from the balance" width="100%"> Remove the centrifuge tube from the balance</td>
 <td align="center"><img src="./docs/gif/Place%20the%20pipette%20on%20the%20pipette%20stand.gif" alt="Place the pipette on the pipette stand" width="100%"> Place the pipette on the pipette stand</td>
 <td align="center"><img src="./docs/gif/Place_conical_flask_on_orbital_shaker.gif" alt="Place conical flask on orbital shaker" width="100%"> Place conical flask on orbital shaker</td>
 </tr>
</table>

| Sample and Cultureware | Equipment Operation | Placement and Relocation |
|---|---|---|
| Pick up the test tube | Close the centrifuge lid | Place the centrifuge tube on the balance |
| Pick up the pipette | Open the centrifuge lid | Remove the centrifuge tube from the balance |
| Remove the petri dish from the incubator | Open the water bath lid | Place the pipette on the pipette stand |
| Place the petri dish in the incubator | Close the spectrophotometer lid | Place conical flask on orbital shaker|

## Getting Started

### Before you begin

Run commands from this repository root. The simulation scripts require an Isaac Sim / Isaac Lab environment; conversion, training, and policy serving require a separate LeRobot environment. The example commands below target Linux x86_64 with an NVIDIA GPU.

You can check the built-in task and robot registries after installing the simulation environment:

```bash
python Data/Keyboard_collection.py --list_tasks --list_robots
```

The included task presets reference the scene assets in `Asset/Scene/` and write datasets under `datasets/` by default. Use `Data/task_registry.py` to inspect or customize task-specific USD paths, camera settings, and success criteria.

## Installation

### 1. Create the environments

Pipette uses two separate Conda environments:

- `env_isaaclab`: NVIDIA Isaac Sim 5.1.0 and Isaac Lab 2.3.2
- `lerobot`: LeRobot with ACT, SmolVLA, and PI0 support

Keeping the simulation and policy environments separate avoids Python dependency conflicts.

The following commands target Linux x86_64. Install Isaac Sim and Isaac Lab first, then LeRobot, and clone Pipette last.

#### Install Isaac Sim 5.1 and Isaac Lab 2.3

Isaac Lab 2.3 provides an official pip package that installs the Isaac Lab packages together with Isaac Sim 5.1:

```bash
# Install the Isaac Sim and Isaac Lab environment.
conda create -y -n env_isaaclab python=3.11
conda activate env_isaaclab
python -m pip install --upgrade pip

python -m pip install torch==2.7.0 torchvision==0.22.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install "isaaclab[isaacsim,all]==2.3.2" --extra-index-url https://pypi.nvidia.com
python -m pip install h5py pyzmq
```

#### Install LeRobot

Create a separate Python 3.12 environment, clone the official LeRobot repository, and install it in editable mode:

```bash
# Install the LeRobot environment.
conda create -y -n lerobot python=3.12
conda activate lerobot
python -m pip install --upgrade pip
conda install -c conda-forge "ffmpeg=6.1.1"

# Run the following commands in the directory where LeRobot will be installed.
git clone https://github.com/huggingface/lerobot.git
cd lerobot
pip install -e .

# Additional dependencies used by Pipette's policy and communication scripts.
pip install transformers accelerate peft
pip install num2words
pip install pyzmq
pip install h5py
```

#### Install Pipette

```bash
# Run the following commands in the directory where Pipette will be installed.
git clone https://github.com/hbhuiyou/Pipette.git
cd Pipette
```

Pipette is cloned last. Keep the terminal in the Pipette repository root when running the following commands so that relative paths such as `Agent/...`, `Data/...`, and `Server/...` resolve correctly.

The Agent clears `PYTHONHOME` and `PYTHONPATH` when launching LeRobot commands to further prevent environment conflicts.

### 2. Configure local paths

```bash
cp Agent/local_config.example.env Agent/local_config.env
```

Then configure the relevant values:

```text
LEROBOT_PYTHON="/path/to/lerobot/python"
LEROBOT_MODEL_ROOT="/path/to/models"
AGENT_ENV_TEMPLATE_USD="/path/to/lab.usd"
AGENT_ASSET_DIR="/path/to/assets"
TENCENTCLOUD_SECRET_ID="your_secret_id"
TENCENTCLOUD_SECRET_KEY="your_secret_key"
TENCENTCLOUD_REGION="ap-guangzhou"
DEEPSEEK_API_KEY=""
DEEPSEEK_BASE_URL="https://api.deepseek.com"
DEEPSEEK_MODEL="deepseek-v4-pro"
```

Do not commit API keys or cloud credentials stored in `Agent/local_config.env`.

## Quick Start

### Natural-language web interface

```bash
python Agent/web_agent.py
```

Open [http://127.0.0.1:7860](http://127.0.0.1:7860), then continue with the desired operation.

The Agent can also replay and inspect datasets, delete selected demonstrations, register a task from a USD scene, and start environment setup. See [`Agent/README.md`](Agent/README.md) for configuration, web controls, intent parsing, and the optional Tencent Hunyuan3D integration.

## Data Pipeline

### 1. Collect demonstrations

```bash
python Data/Keyboard_collection.py \
  --task_id pick_up_the_tube \
  --num_demos 30 \
  --dataset_file ./datasets/pick_up_the_tube.hdf5
```

Keyboard controls:

- `R`: save the current demonstration
- `SPACE`: skip the current demonstration
- `P`: stop collection

### 2. Inspect dataset

```bash
python Data/inspect_hdf5_dataset.py \
  --file ./datasets/pick_up_the_tube.hdf5 \
  --show-attrs
```

### 3. Generate simulation-augmented data

```bash
python Data/Generate_data.py \
  --task_id pick_up_the_tube \
  --dataset_file ./datasets/pick_up_the_tube.hdf5 \
  --output_file ./datasets/pick_up_the_tube_aug.hdf5 \
  --num_envs 3 \
  --light_intensity_scales 0.8 \
  --temporal_speed_scales 1.2 \
  --camera_jitter_count 5 \
  --include_original \
  --headless
```

The augmentation workers re-execute trajectories and regenerate observations instead of editing saved images. The current pipeline supports:

- lighting-intensity perturbation;
- temporal speed perturbation and trajectory resampling;
- top, main, and wrist camera pose perturbation;
- bounded joint-action noise;

### 4. Convert HDF5 to LeRobot

Run this command in the LeRobot environment:

```bash
python Data/hdf5_to_lerobot.py \
  --hdf5-path ./datasets/pick_up_the_tube_aug.hdf5 \
  --repo-id pick_up_the_tube \
  --output-dir /absolute/path/to/lerobot/pick_up_the_tube \
  --fps 10 \
  --frame-filter fresh \
  --stride 3
```

The converter writes three RGB features, robot state, actions, and the language instruction. It infers image resolution and state/action widths from the input HDF5 file. Frames without a fresh visual observation are filtered to preserve temporal alignment.

## Policy Training

For an initial training run, use a single dataset to train SmolVLA in the LeRobot environment:

```bash
# Run from the LeRobot directory.
lerobot-train \
  --dataset.repo_id=/path/to/lerobot/datasets/pick_tube \
  --policy.type=smolvla \
  --policy.repo_id=local/smolvla_pick_tube \
  --output_dir=/path/to/checkpoints/pick_tube_smolvla \
  --batch_size=8 \
  --steps=20000 \
  --wandb.enable=false
```

Replace the dataset and model output paths with actual absolute paths. This example trains only the `pick_tube` task and is useful for validating the dataset, GPU memory, and training environment before batch training.

The batch entry point supports ACT, SmolVLA, and PI0, with Franka as the default robot arm:

```bash
python run_lerobot_batch_train.py \
  --model smolvla \
  --dataset-version aug \
  --dataset-root /path/to/lerobot/datasets \
  --output-root /path/to/checkpoints
```

Use `--dry-run` to inspect all generated `lerobot-train` commands before starting:

```bash
python run_lerobot_batch_train.py --model pi0 --dataset-version raw --dry-run
```

The manuscript uses the following training configuration:

| Policy | Batch size | Training steps | Additional settings |
|---|---:|---:|---|
| ACT | 32 | 15,000 | Default precision |
| SmolVLA | 8 | 20,000 | Default precision |
| PI0 | 4 | 20,000 | BF16, frozen vision encoder, expert-only training, gradient checkpointing |

## Closed-Loop Evaluation

Evaluation uses a LeRobot policy server and an Isaac Lab client connected through ZMQ.

Start the policy server in the LeRobot environment:

```bash
python Server/server_brain.py \
  --policy-path /path/to/checkpoint \
  --policy-type smolvla \
  --bind tcp://127.0.0.1:5555 \
  --device cuda
```

Start the matching Isaac Lab client:

```bash
python Client/inference_smolvla.py \
  --task-id pick_up_the_tube \
  --server-endpoint tcp://127.0.0.1:5555 \
  --episodes 100 \
  --output-json ./outputs/smolvla_pick_up_the_tube.json
```

Available clients:

- `Client/inference_act.py`
- `Client/inference_smolvla.py`
- `Client/inference_pi0.py`

Each episode records success or failure, failure reason, runtime, policy and control frequencies, and task-specific evaluator metrics.

## Limitations

- The task presets and success thresholds are tuned primarily for the included Franka Panda scenes. The active robot registry also includes Nero Gripper D435 and UFactory xArm7; users should validate each task after changing the robot or scene.
- Systematic real-robot validation and sim-to-real transfer are future work.
- Success evaluators are currently task-specific and threshold-based.
- Language-guided task registration still requires users to verify USD paths, prim paths, camera settings, and success thresholds.

## Acknowledgements

Pipette builds on [NVIDIA Isaac Sim](https://developer.nvidia.com/isaac-sim), [Isaac Lab](https://isaac-sim.github.io/IsaacLab/), and [LeRobot](https://github.com/huggingface/lerobot).

Pipette

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README