EgoGazeVQA-Plus

KO | EN

gitlite — search

#javascript #hacktoberfest #react #golang #android #machine-learning #linux #docker #nodejs #java #awesome #cli #kubernetes #mcp

EgoGazeVQA-Plus

(★ 14)

Official code for EgoGazeVQA, accepted to NeurIPS D&B 2025

EgoGazeVQA-Plus 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting

[![Paper](https://img.shields.io/badge/arXiv-2509.07447-b31b1b.svg)](https://arxiv.org/abs/2509.07447)
[![Project Page](https://img.shields.io/badge/Project-Page-green)](https://taiyi98.github.io/projects/EgoGazeVQA)
[![Dataset](https://img.shields.io/badge/🤗-Dataset-yellow)](https://huggingface.co/datasets/taiyi09/EgoGazeVQA)

> **Taiying Peng1, Jiacheng Hua2, Miao Liu2†, Feng Lu1†** 
> 1State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University 
> 2College of AI, Tsinghua University 
> **NeurIPS D&B 2025**

---
The preliminary EgoGazeVQA dataset is currently available. The expanded EgoGazeVQA+ annotations, verified samples, and official data splits are being prepared for release. **"EgoGazeVQA+: Benchmarking Egocentric Video Intent Understanding with Agentic Verification and Gaze-Guided Attention Intervention"**.
## Overview

This repository provides the official code for **EgoGazeVQA**, a benchmark for evaluating multimodal large language models (MLLMs) on egocentric video understanding tasks with gaze guidance.

**Code Purpose:**
- Generate gaze-guided QA pairs from egocentric videos (spatial, temporal, causal intent questions)
- Evaluate MLLMs with three gaze-guided prompting strategies (textual, visual marks, salience maps)
- Calculate and analyze model performance on intent understanding tasks

## Timeline

- **[2025-10]** Code and dataset publicly released
- **[2025-09]** Paper accepted at NeurIPS 2025 D&B Track

## TODO

- [ ] Add fine-tuning scripts for LoRA adaptation

## Repository Structure

```
EgoGazeVQA/
├── generate_tool/
│   ├── auto.sh
│   ├── spatial.py
│   ├── temporal.py
│   ├── causal.py
│   └── create_datasets.py
├── test_tool/
│   ├── qwenvl_test/
│   │   ├── test_wo.py
│   │   ├── test_gaze.py
│   │   ├── test_mark.py
│   │   └── test_saliencemap.py
│   ├── prompt_gazees/
│   ├── multiframes/
│   ├── gaze_trajectory.py
│   └── caculate.py
```

## Installation

```bash
git clone https://github.com/taiyi98/EgoGazeVQA.git
cd EgoGazeVQA

conda create -n egogazevqa python=3.10
conda activate egogazevqa

pip install -r requirements.txt
```

## Dataset

Download from [🤗 Hugging Face](https://huggingface.co/datasets/taiyi09/EgoGazeVQA):

```bash
huggingface-cli download taiyi98/EgoGazeVQA --repo-type dataset --local-dir ./data
```

## Usage

### Generate QA Pairs

```bash
cd generate_tool

# Generate for specific video
python spatial.py --video_id <VIDEO_ID> --target_index <INDEX>
python temporal.py --video_id <VIDEO_ID> --target_index <INDEX>
python causal.py --video_id <VIDEO_ID> --target_index <INDEX>

# Batch processing
bash auto.sh
```

### Evaluate Models

```bash
cd test_tool/qwenvl_test

python test_wo.py              # Baseline (no gaze)
python test_gaze.py            # Textual gaze prompt
python test_mark.py            # Visual gaze marks
python test_saliencemap.py     # Gaze salience maps
```

### Calculate Results

```bash
cd test_tool
python caculate.py --result_file <RESULT_CSV_PATH>
```

## Citation

```bibtex
@article{peng2026eye,
  title={In the eye of mllm: Benchmarking egocentric video intent understanding with gaze-guided prompting},
  author={Peng, Taiying and Hua, Jiacheng and Liu, Miao and Lu, Feng},
  journal={Advances in Neural Information Processing Systems},
  volume={38},
  year={2026}
}
```

## License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

EgoGazeVQA-Plus

EgoGazeVQA-Plus 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

EgoGazeVQA-Plus

EgoGazeVQA-Plus 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README