Manual2SkillPP_official

KO | EN

gitlite — search

#java #linux #open-source #ai #codex #kubernetes #rust #list #react-native #gemini

Manual2SkillPP_official

(★ 9)

No description available.

파일 탐색기

ZIP 다운로드

- - cover_image.png
- - __init__.py
  - assembly.py
  - blender_build_scene_minimal.py
  - compare_pair.py
  - data_loader.py
  - extract_gt_from_blend.py
  - flip_analyzer.py
  - orchestrator_utils.py
  - part.py
  - random_transformor.py
  - README.md
  - run.py
  - run_blender.ps1
  - run_eval.ps1
- - ablation_blank_page.png
  - dataset_stats.py
  - evaluation.py
  - print_final_results.py
  - random_sampler.py
  - run_dashboard.sh
  - run_inference.py
  - run_vlm.sh
  - runner.py
  - utils.py
- .env.example
- .gitignore
- environment.yml
- README.md
- requirements.txt
- README.md
- README_tasks.md
.gitignore
README.md
REVIEW_NOTES.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<a href="https://crtie.github.io/">Chenrui Tie</a>,
 <a href="https://owensun2004.github.io/">Shengxiang Sun</a>,
 <a href="https://scholar.google.com/citations?user=jf2Z6QQAAAAJ&hl=en">Yudi Lin</a>,
 <a href="https://scholar.google.com/citations?user=UqaREo8AAAAJ&hl=zh-CN">Yanbo Wang</a>,
 Zhongrui Li,
 Zhouhan Zhong,
 <a href="https://www.linkedin.com/in/jinxuan-zhu-08a8972b7/">Jinxuan Zhu</a>,
 Yiman Pang,
 <a href="https://github.com/chenhn02">Haonan Chen</a>,
 <a href="https://sgtvincent.github.io/">Junting Chen</a>,
 <a href="https://warshallrho.github.io/">Ruihai Wu</a>,
 <a href="https://linsats.github.io/">Lin Shao</a>

ICRA 2026

![Manual2Skill++ overview](https://raw.githubusercontent.com/crtie/Manual2SkillPP_official/main/high_level/assets/cover_image.png)

Manual2Skill++ turns visual instruction manuals into connector-aware robot assembly skills. The system first uses vision-language models to infer assembly order and connector choices from manual pages and part observations. It then converts those connector-level predictions into step-wise 6D part poses, and uses low-level simulation environments to study executable assembly behaviors.

This release contains the two main code components used in the paper:

- `high_level/`: manual understanding, VLM planning, hierarchical graph extraction, and connector-aware pose alignment.
- `isaac_lab/`: Isaac Lab task overlay for low-level Franka assembly environments.

The two components can be used independently. Most users should start with `high_level/`, which does not require Isaac Sim. Use `isaac_lab/` only when you want to run the simulator tasks.

## Repository Layout

```text
.
  high_level/
    VLM_Planning/          # prompts, VLM clients, dataset loader, graph extraction, dashboard
    pose_alignment/        # connector constraints and geometric pose solver
    assets/                # README and project images
    environment.yml        # conda environment for the high-level pipeline
    requirements.txt       # Python dependencies used by the high-level pipeline
  isaac_lab/
    source/isaaclab_tasks/isaaclab_tasks/direct/np/
                            # task package, task configs, assets, and controllers
    scripts/environments/teleoperation/
                            # keyboard teleoperation entry point
    README_tasks.md         # task ids, task_idx ranges, and subtask descriptions
```

The high-level release includes 21 instruction-manual datasets under `VLM_Planning/dataset/`, 18 normalized pose-alignment assemblies under `pose_alignment/data/`, and two curated end-to-end dashboard examples: `sundvik` and `gear`.

The low-level release includes Chair, Vasskar, Plane, and Lego assembly tasks. The simulator code is an overlay for Isaac Lab 4.5, not a full Isaac Lab checkout.

## Pipeline Overview

Manual2Skill++ has three practical stages:

1. **VLM planning.** Manual pages and scene images are converted into step-level connector choices and a hierarchical assembly graph.
2. **Pose alignment.** Part definitions, connector geometry, and ordered connector constraints are solved into 6D part poses for each assembly step.
3. **Low-level simulation.** Franka assembly environments provide the interaction layer for teleoperation and policy development.

The dashboard connects the first two stages for inspection. The Isaac Lab tasks are provided separately so users who only need manual parsing or pose alignment do not need to install the simulator stack.

## High-Level Setup

Create the Python environment from the high-level directory:

```bash
cd high_level
conda env create -f environment.yml
conda activate manualpp
```

Optional Blender-based mesh evaluation requires a local Blender installation. Blender's `bpy` and `mathutils` modules are provided by Blender itself and are not installed by `environment.yml`.

## Model Credentials

VLM inference requires credentials for the backend you choose:

```bash
export OPENAI_API_KEY="..."
export GEMINI_API_KEY="..."
export DMX_API_KEY="..."
```

For OpenAI-compatible relay services, set:

```bash
export OPENAI_BASE_URL="https://your-compatible-endpoint/v1"
export OPENAI_MODEL="gpt-4o"
```

The dashboard can also read credentials entered in the UI for the current server process. See `high_level/.env.example` for the complete list of supported environment variables.

## Interactive Dashboard

The dashboard is the easiest way to understand the high-level pipeline. It shows the manual pages, prompts, VLM responses, parsed connector predictions, graph view, and pose-alignment visualization in one interface.

```bash
cd high_level
conda activate manualpp
./VLM_Planning/run_dashboard.sh
```

Open:

```text
http://127.0.0.1:7860
```

In the dashboard, select an assembly from `VLM_Planning/dataset/`, run or inspect a VLM prediction, and review the structured connector decisions. For `sundvik` and `gear`, you can also select an assembly step and visualize the connected parts after pose alignment.

For pose visualization, the dashboard supports two hole sources:

- `Ground truth`: use annotated hole pairs from the dataset.
- `Latest predictions`: use the most recent dashboard VLM run if that run contains the selected step.

## Headless VLM Planning

Use the shell entry point when you want reproducible command-line VLM runs without the dashboard:

```bash
cd high_level
conda activate manualpp
./VLM_Planning/run_vlm.sh \
  --model gpt \
  --assembly_name applaro \
  --print_prompt \
  --print_comparison \
  --no_dashboard
```

Supported `--model` values:

- `gpt`: OpenAI SDK, uses `OPENAI_API_KEY`.
- `gemini`: Google GenAI SDK, uses `GEMINI_API_KEY`.
- `gpt_dmx`: DMX REST endpoint, uses `DMX_API_KEY`.
- `gemini_dmx`: DMX REST endpoint, uses `DMX_API_KEY`.

Useful debugging flags:

- `--max_steps N`: run only the first `N` steps.
- `--print_prompt`: save prompts.
- `--print_response`: save raw model responses.
- `--print_comparison`: save prediction-vs-ground-truth comparisons.
- `--run_ablation`: blank one manual page step for ablation experiments.

Regular outputs are written under:

```text
high_level/VLM_Planning/dataset/<assembly>/results/
```

Saved paper-table snapshots are stored under each assembly's `results_table_3/` directory.

## Pose Alignment

The pose-alignment module can be run without any VLM calls. It uses normalized JSON definitions for parts, connectors, and ordered assembly constraints, then writes a complete pose state for each assembly step.

List available pose-alignment assemblies:

```bash
cd high_level
conda activate manualpp
python -m pose_alignment.run --list
```

Run the curated examples:

```bash
python -m pose_alignment.run sundvik gear \
  --output pose_alignment/results \
  --save-steps \
  --no-randomize
```

Common options:

```text
--all run all assembly definitions
--repeat N number of random restarts per assembly
--save-steps save assembly_step_<n>_complete.json files
--no-randomize start from the poses stored in the data files
--translation-range MM random translation range
--euler-range DEG random rotation range
--quiet reduce outer CLI logs
```

Important outputs:

```text
high_level/pose_alignment/results/<assembly>/run_/
 assembly_step_<n>_complete.json
 assembly_FINAL_state.json
```

Each state file stores `parts_state[part_id].transform_matrix`, a 4x4 row-major transform matrix with translations in millimeters. Mesh-based evaluation against Blender scenes is documented in `high_level/pose_alignment/README.md`.

## Evaluation Utilities

Print saved paper-table results:

```bash
cd high_level
conda activate manualpp
python VLM_Planning/print_final_results.py --use_table_3_results
```

Evaluate regular VLM runs:

```bash
python VLM_Planning/print_final_results.py --model gpt --type regular
```

Compute dataset statistics:

```bash
python VLM_Planning/dataset_stats.py --base-dir VLM_Planning/dataset
```

## Low-Level Isaac Lab Environments

Install Isaac Lab 4.5 separately first. Then copy or symlink the Manual2Skill++ task overlay into your Isaac Lab checkout.

From the root of this repository:

```bash
cp -r isaac_lab/source/isaaclab_tasks/isaaclab_tasks/direct/np \
  /path/to/IsaacLab/source/isaaclab_tasks/isaaclab_tasks/direct/

cp isaac_lab/scripts/environments/teleoperation/teleop_se3_agent_custom.py \
  /path/to/IsaacLab/scripts/environments/teleoperation/
```

The overlay registers these task families:

- Chair: `Isaac-Franka-Chair1-Direct-v0` through `Isaac-Franka-Chair6-Direct-v0`
- Vasskar: `Isaac-Franka-Vasskar1-Direct-v0` and `Isaac-Franka-Vasskar2-Direct-v0`
- Plane: `Isaac-Franka-Plane1-Direct-v0` through `Isaac-Franka-Plane4-Direct-v0`
- Lego: `Isaac-Franka-Lego1-Direct-v0` through `Isaac-Franka-Lego7-Direct-v0`

Detailed `task_idx` ranges and subtask descriptions are in `isaac_lab/README_tasks.md`.

Run the low-level tasks from your Isaac Lab checkout:

```bash
cd /path/to/IsaacLab
conda activate isaaclab
python scripts/environments/teleoperation/teleop_se3_agent_custom.py \
  --task Isaac-Franka-Lego1-Direct-v0 \
  --task_idx 1 \
  --num_envs 1 \
  --teleop_device keyboard \
  --sensitivity 10
```

Use `--task_idx <N>` to select an internal subtask at launch time. For a non-interactive smoke test:

```bash
python scripts/environments/teleoperation/teleop_se3_agent_custom.py \
  --task Isaac-Franka-Lego1-Direct-v0 \
  --task_idx 1 \
  --num_envs 1 \
  --teleop_device keyboard \
  --sensitivity 10 \
  --headless \
  --max_steps 5
```

In headless mode, the teleoperation script uses zero actions if the Isaac Sim keyboard window is unavailable. Real keyboard teleoperation should be run without `--headless`.

If custom gym ids are not visible in your own scripts, import the task package before calling `gym.make(...)`:

```python
import isaaclab_tasks.direct.np
```

The provided teleoperation script already performs this import.

## More Documentation

- `high_level/README.md`: high-level planning, dashboard, headless VLM runs, and evaluation utilities.
- `high_level/pose_alignment/README.md`: pose-alignment data format, solver options, and Blender evaluation.
- `isaac_lab/README.md`: concise Isaac Lab overlay installation guide.
- `isaac_lab/README_tasks.md`: complete task id and `task_idx` reference.

## Citation

If you find this code useful, please cite:

```bibtex
@article{tie2025manual2skillpp,
  title={Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models},
  author={Tie, Chenrui and Sun, Shengxiang and Lin, Yudi and Wang, Yanbo and Li, Zhongrui and Zhong, Zhouhan and Zhu, Jinxuan and Pang, Yiman and Chen, Haonan and Chen, Junting and Wu, Ruihai and Shao, Lin},
  journal={arXiv preprint arXiv:2510.16344},
  year={2025}
}
```

Manual2SkillPP_official

오너의 다른 레포

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Manual2SkillPP_official

오너의 다른 레포

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README