VideoChat-R1 - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #go #machine-learning #linux #docker #nodejs #awesome #cli #kubernetes #awesome-list #pytorch #openai #open-source

VideoChat-R1

(★ 268)

[NIPS2025] VideoChat-R1 & R1.5: Enhancing Spatio-Temporal Perception and Reasoning via Reinforcement Fine-Tuning

File Explorer

Download Latest Version (.zip)

- - ddp.yaml
  - zero2.yaml
  - zero3.yaml
- - .gitignore
  - .pre-commit-config.yaml
  - LICENSE
  - pyproject.toml
  - README.md
  - setup.py
- - __init__.py
- - __init__.py
  - data_config.py
  - eval_prompts.py
  - evaluate_cls_quality_c8.py
  - evaluate_gqa.py
  - evaluate_grounding.py
  - evaluate_qa.py
  - evaluate_track.py
  - my_qwen_utils.py
- - run_grpo_video_cls_qa.sh
  - run_grpo_video_gqa.sh
  - run_grpo_video_gqa_nothink_3e.sh
  - run_grpo_video_qa.sh
  - run_grpo_video_qa_nothink.sh
  - run_grpo_video_task.sh
  - run_grpo_video_tg.sh
  - run_sft_video_cls_qa.sh
  - run_sft_video_gqa.sh
  - run_sft_video_track.sh
  - zero3_offload.json
- - grpo_tasks.sh
  - zero2_offload.json
  - zero3.json
  - zero3.yaml
  - zero3_offload.json
- - __init__.py
  - eval_longvideo.py
  - eval_lvbench.py
  - eval_mlvu.py
  - eval_mvbench.py
  - eval_videomme.py
  - eval_videommmu.py
  - eval_vsi.py
  - evaluate_anet.py
  - evaluate_cgbench.py
  - evaluate_chara.py
  - evaluate_det.py
  - evaluate_gqa.py
  - evaluate_qvh.py
  - evaluate_rextime.py
  - evaluate_tracking.py
  - my_vision_process.py
framework.png
perception.jpg
perception.png
README.md
requirements.txt
sotas.png

# Installation Guide

1. Get the code

git clone https://github.com/OpenGVLab/VideoChat-R1

Downloads the entire project code from GitHub to your computer.

cd VideoChat-R1

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

jupyter notebook

Launches Jupyter in your browser so you can open and run the notebook (.ipynb) files.

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# VideoChat-R1 & -R1.5: Spatio-Temporal RL for Video Perception and Reasoning

## :fire: Updates

- [x] **2025/09/26**:🔥🔥🔥 We release our VideoChat-R1.5 model at [Huggingface](https://huggingface.co/OpenGVLab/VideoChat-R1_5), [paper](https://arxiv.org/pdf/2509.21100), and eval code.
- [x] **2025/09/22**: 🎉🎉🎉 Our VideoChat-R1.5 is accepted by NIPS2025.
- [x] **2025/04/22**:🔥🔥🔥 We release our VideoChat-R1-caption at [Huggingface](https://huggingface.co/collections/OpenGVLab/videochat-r1-67fbe26e4eb08c83aa24643e).
- [x] **2025/04/14**:🔥🔥🔥 We release our VideoChat-R1 and  VideoChat-R1-thinking at [Huggingface](https://huggingface.co/collections/OpenGVLab/videochat-r1-67fbe26e4eb08c83aa24643e).
- [x] **2025/04/10**:🔥🔥🔥 We release our VideoChat-R1 [paper](https://arxiv.org/abs/2504.06958) and code.

## 🎯 Performances on Video Benchmarks

![alt text](https://raw.githubusercontent.com/OpenGVLab/VideoChat-R1/main/sotas.png)

Across short-form & long-form videos, temporal grounding, video reasoning, and spatio-temporal perception, the model delivers consistently stronger results.

## :parrot: Introduction

![alt text](https://raw.githubusercontent.com/OpenGVLab/VideoChat-R1/main/framework.png)

We adopt multi-task joint RL to strengthen the model’s spatio-temporal perception and reasoning capabilities.

![alt text](https://raw.githubusercontent.com/OpenGVLab/VideoChat-R1/main/perception.jpg)

During inference, we simulate hierarchical human attention to enable the model to progressively localize the Region of Interest (ROI) within input videos. This multi-step perception process ensures that the model's performance improves with each step.

## Demo & Inference

Please refer to [hf README](https://huggingface.co/OpenGVLab/VideoChat-R1_7B) for the steps required to perform inference..

## Evaluation

See [eval_scripts](eval_scripts) and [lmms-eval_videochat](lmms-eval_videochat).

## Training

See [training_scripts](training_scripts).

# :page_facing_up: Citation

If you find this project useful in your research, please consider cite:
```BibTeX
@article{li2025videochatr1,
  title={VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning},
  author={Li, Xinhao and Yan, Ziang and Meng, Desen and Dong, Lu and Zeng, Xiangyu and He, Yinan and Wang, Yali and Qiao, Yu and Wang, Yi and Wang, Limin},
  journal={arXiv preprint arXiv:2504.06958},
  year={2025}
}

@article{yan2025videochatr15,
  title={VideoChat-R1.5: Visual Test-Time Scaling to Reinforce Multimodal Reasoning by Iterative Perception},
  author={Yan, Ziang and Li, Xinhao and He, Yinan and Zhengrong Yue and Zeng, Xiangyu and Wang, Yali and Qiao, Yu and Wang, Limin and Wang, Yi},
  journal={arXiv preprint arXiv:2509.21100},
  year={2025}
}
```

For any inquiries regarding this work, please contact us at yanziang@pjlab.org.cn .

<!-- # :dizzy: Acknowledgement

Thanks to the open source of the following projects: [Qwen](https://github.com/QwenLM/Qwen), [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval), their implementation provides valuable reference experience for our project. -->