agenthoi

KO | EN

gitlite — search

#sql #large-language-models #windows #csharp #mysql #tensorflow #artificial-intelligence #gpt #command-line #typescript

(★ 9)

AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

파일 탐색기

ZIP 다운로드

- demo_thumbnail.png
- hoi_teaser.png
- method_overview.png
- train_t2m_480p_1_3b.yaml
- train_wanx_480p_hoi.yaml
- - __init__.py
  - agent_refiner.py
  - prompts.py
- - __init__.py
  - image2video_dataset.py
  - text2video_dataset.py
- - __init__.py
  - scheduling_flow_match_discrete.py
- - __init__.py
  - communication.py
  - data_utils.py
  - diffusion_utils.py
  - fsdp_utils.py
  - load.py
  - model_utils.py
  - parallel_states.py
  - torch_utils.py
- - __init__.py
  - first_last_frame2video.py
  - image2video.py
  - text2video.py
- __init__.py
- arguments.py
- constants.py
- - example.json
  - README.md
- README.md
- - install_main.sh
- - inference_wanx.py
- - preprocess_agent_refine.py
  - preprocess_agent_refine.sh
- - train_wanx.py
  - train_wanx.sh
  - train_wanx_mul.sh
  - train_wanx_nodes.sh
.gitignore
download_models.sh
README.md
requirements.txt
run_inference.sh
run_train.sh
setup.py

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

[![arXiv](https://img.shields.io/badge/arXiv-2607.22241-b31b1b)](https://arxiv.org/abs/2607.22241)
[![Model](https://img.shields.io/badge/Model-HuggingFace-F9D648)](https://huggingface.co/ProAudience/agenthoi)

</div>

**AgentHOI** is a *thinking-before-generation* framework in which specialized VLM agents turn person, object, and text inputs into a physically grounded action plan for a HOI video generator. During training, a frozen text-to-motion teacher provides token-relation distillation targets; it is removed at inference for motion-free image-and-text-conditioned generation.

## Video Demo

## Method Overview

## Quick Start

The demo expects Linux, Python 3.10, ffmpeg, and large GPUs. A single 80GB GPU should run the default demo with CPU offload; multi-GPU inference is faster but not required. End-to-end inference uses InternVL3.5-38B and Wan2.1-I2V-14B.

1. Create the environment.

```bash
conda create -n agenthoi python=3.10 -y
conda activate agenthoi

conda install -c conda-forge ffmpeg

pip install "torch>=2.4.0" torchvision --index-url https://download.pytorch.org/whl/cu124

pip install flash_attn==2.7.4.post1 --no-build-isolation
pip install -r requirements.txt
pip install "xfuser>=0.4.0"
```

2. Download the model weights.

By default, weights are downloaded under `checkpoints/agenthoi`. Set `DOWNLOAD_ROOT` if you want another location.

```bash
bash download_models.sh
```

This downloads:

| Repo | Used as |
| --- | --- |
| [`Wan-AI/Wan2.1-I2V-14B-480P`](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) | Wan2.1-I2V backbone for video generation |
| [`OpenGVLab/InternVL3_5-38B`](https://huggingface.co/OpenGVLab/InternVL3_5-38B) | VLM for the four reasoning agents |
| [`ProAudience/agenthoi`](https://huggingface.co/ProAudience/agenthoi) | AgentHOI checkpoints, including `t2v/checkpoint-9500` |

The default layout is:

```text
checkpoints/agenthoi/
|-- Wan2.1-I2V-14B-480P/
|-- InternVL3_5-38B/
`-- agenthoi/
    |-- t2v/checkpoint-9500/
    `-- t2m/checkpoint-8000/
```

`TRANSFORMER_PATH` should point to the inner `agenthoi/t2v/checkpoint-9500` directory. The `t2m/checkpoint-8000` checkpoint is training-only.

3. Run the demo.

```bash
bash run_inference.sh
```

Outputs are written to `outputs/demo/`. The intermediate agent-refined timeline is saved under `outputs/demo/agent_refined/`.

For a single 80GB GPU, run one process in each stage:

```bash
AGENT_WORLD_SIZE=1 NPROC_PER_NODE=1 bash run_inference.sh
```

You can also override individual paths:

```bash
WAN_BASE_PATH=/path/to/Wan2.1-I2V-14B-480P \
TRANSFORMER_PATH=/path/to/agenthoi/t2v/checkpoint-9500 \
INTERNVL_PATH=/path/to/InternVL3_5-38B \
bash run_inference.sh
```

## Inference

`run_inference.sh` runs two stages:

1. **Agent-based HOI reasoning**: four VLM agents refine the prompt into a second-by-second interaction timeline.
2. **Video generation**: a Wan2.1-I2V-14B based model generates the HOI video from the refined timeline, person image, and object image.

The bundled examples live in `examples/I2V/example.json`.

```bash
SKIP_AGENT=1 DATASET_PATH=path/to/refined.json bash run_inference.sh
```

Extra arguments after `run_inference.sh` are forwarded to
`scripts/inference/inference_wanx.py` and override the corresponding defaults:

```bash
bash run_inference.sh --sample_steps 30
```

## Use Your Own Inputs

Add a case to `examples/I2V/example.json`:

```json
{
  "image_id": "my_case",
  "caption": "A person picks up a mug and sips from it.",
  "ref_image_path": "examples/I2V/assets/humans/my_case.png",
  "ref_obj_image_path": "examples/I2V/assets/objects/my_case.png"
}
```

Then run:

```bash
bash run_inference.sh
```

The inputs are:

| Field | Description |
| --- | --- |
| `image_id` | Unique id used as the output filename stem |
| `caption` | Coarse text prompt; the agents refine it into a timeline |
| `ref_image_path` | Reference person image |
| `ref_obj_image_path` | Reference object image, used by both the agent and video generator |

## Training

Training is launched through `run_train.sh` and uses two configs:

- `configs/train_t2m_480p_1_3b.yaml` trains `M_motion`, the auxiliary text-to-motion teacher used to provide TRD targets.
- `configs/train_wanx_480p_hoi.yaml` trains `M`, the final HOI video generator, with `M_motion` frozen for token-relation distillation.

Training data is not included in this release; prepare your own data and set the dataset paths in the selected YAML before launching training.

Train `M_motion` first:

```bash
bash run_train.sh motion
```

Then train `M`:

```bash
bash run_train.sh video
```

To download the additional Wan2.1-T2V-1.3B backbone needed for retraining `M_motion`, run:

```bash
DOWNLOAD_TRAIN=1 bash download_models.sh
```

## Citation

If AgentHOI is useful for your research, please cite:

```bibtex
@misc{huang2026agenthoimultiagentreasoninghumanobjectinteraction,
      title={AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment}, 
      author={Ziyao Huang and Shunkai Li and Juan Cao and Chenyu Li and Youliang Zhang and Zixiang Zhou and Cong Wang and Yuan Zhou and Qinglin Lu and Fan Tang},
      year={2026},
      eprint={2607.22241},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.22241}, 
}
```

## Related Work

You can find our last work here: [HOMA](https://github.com/bone-11/homa).

## Acknowledgements

AgentHOI builds on open research from [Wan2.1](https://github.com/Wan-Video/Wan2.1), [InternVL](https://github.com/OpenGVLab/InternVL), [diffusers](https://github.com/huggingface/diffusers), and [FlashAttention](https://github.com/Dao-AILab/flash-attention).

agenthoi

파일 탐색기

# CDN으로 사용하기

Was this content helpful?

Recent Feedback

Download README