ZS-F-VQA - GitLite

KO | EN

gitlite — search

#hacktoberfest #typescript #llm #go #golang #android #docker #nodejs #windows #awesome #cli #kubernetes #openai #open-source

ZS-F-VQA

(★ 72)

[Paper][ISWC 2021] Zero-shot Visual Question Answering using Knowledge Graph

파일 탐색기

최종 버전 다운로드 (.zip)

- - .DS_Store
  - __init__.py
  - base.py
  - fvqa.py
  - preprocess.py
- - .DS_Store
  - __init__.py
  - answer_mlp.py
  - answer_net.py
  - attention.py
  - classifier.py
  - counting.py
  - fc.py
  - fusion_ban.py
  - fusion_mlp.py
  - fusion_net.py
  - fusion_san.py
  - fusion_updn.py
  - language_model.py
  - requirements.txt
  - vector.py
- - __init__.py
  - logger.py
  - metric.py
  - module.py
  - utils.py
  - vocab.py
- - .DS_Store
  - __init__.py
  - metrics.py
  - tool.py
- config.py
- deal_data.py
- joint_test.py
- main.py
- run_FVQA.sh
- run_FVQA_train.sh
- run_ZSL.sh
- run_ZSL_train.sh
- - .DS_Store
- .DS_Store
- all_explainable.png
- Model_architecture.png
- README.md
- ZS-F-VQA_Paper_README.md
- ZS-F-VQA_Project_README.md
.DS_Store
LICENSE
README.md
requirement.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/China-UK-ZSL/ZS-F-VQA

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd ZS-F-VQA

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ZS-F-VQA

![](https://img.shields.io/badge/version-1.0.1-blue)
[![license](https://img.shields.io/github/license/mashape/apistatus.svg?maxAge=2592000)](https://github.com/China-UK-ZSL/ZS-F-VQA/blob/main/LICENSE)
[![arxiv badge](https://img.shields.io/badge/arXiv-2107.05348-red)](http://arxiv.org/abs/2107.05348)

[*Zero-shot Visual Question Answering using Knowledge Graph*](https://arxiv.org/abs/2107.05348) [ ISWC 2021 ]

>In this work, we propose a Zero-shot VQA algorithm using knowledge graphs and a mask-based learning mechanism for better incorporating external knowledge, and present new answer-based Zero-shot VQA splits for the F-VQA dataset.

## 🔔 News
- **`2024-02` We preprint our Survey [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](http://arxiv.org/abs/2402.05391)  [[`Repo`](https://github.com/zjukg/KG-MM-Survey)].**

## Model Architecture
![Model_architecture](https://github.com/China-UK-ZSL/ZS-F-VQA/blob/main/figures/Model_architecture.png)

<br />

## Usage

### Requirements
- `python >= 3.5`
- `PyTorch >= 1.6.0`

For more detail of requirements: 
```bash
 pip install -r requirements.txt
```

### Data

Location of 5 **F-VQA** train / test data split:
- ```data/KG_VQA/fvqa/exp_data/train_data```
- ```data/KG_VQA/fvqa/exp_data/test_data```

Location of 5 **ZS-F-VQA** train / test data split: 
- ```data/KG_VQA/fvqa/exp_data/train_seen_data```
- ```data/KG_VQA/fvqa/exp_data/test_unseen_data```

Answers are available at ``data/KG_VQA/data/FVQA/new_dataset_release/.``

**Image:**
- Image folder (put all your `.JPEG`/`.jpg` file here):
```data/KG_VQA/fvqa/exp_data/images/images```
- Image feature: 
  - `fvqa-resnet-14x14.h5` pretrained: [GoogleDrive](https://drive.google.com/file/d/1YG9hByw01_ZQ6_mKwehYiddG3x2Cxatu/view?usp=sharing) or [BaiduCloud](https://pan.baidu.com/s/1ks84AWSXxJJ_7LwnzWdEnQ) (password:16vd)
  -  `fvqa36_imgid2idx.pkl` and `fvqa_36.hdf5` pretrained: [GoogleDrive](https://drive.google.com/file/d/1wfgmPhNF7DR7_yEAr8lxjtdsko7lLCWj/view?usp=sharing) or [BaiduCloud](https://pan.baidu.com/s/11KRiw2jvPBzgd3xUbynHjw?pwd=zsqa) (password:zsqa)
- Origin images are available at [FVQA](https://github.com/wangpengnorman/FVQA) with [download_link](https://www.dropbox.com/s/iyz6l7jhbt6jb7q/new_dataset_release.zip?dl=0).
- Other vqa dataset: you could generate a pretrained image feature via this way ([Guidance](https://github.com/hexiang-hu/answer_embedding/issues/3) / [code](https://github.com/Cyanogenoid/pytorch-vqa/blob/master/preprocess-images.py))
- The generated `.h` file should be placed in :
```data/KG_VQA/fvqa/exp_data/common_data/.```
  
**Answer / Qusetion vocab:**
- The generated file `answer.vocab.fvqa.json` & `question.vocab.fvqa.json`  now are available at :
```data/KG_VQA/fvqa/exp_data/common_data/.```
- Other vqa dataset: code for [process answer vocab](https://github.com/hexiang-hu/answer_embedding/blob/master/tools/preprocess_answer.py) and [process questions vocab](https://github.com/hexiang-hu/answer_embedding/blob/master/tools/preprocess_question.py)

---

### Pretrained Model ([url](https://www.dropbox.com/sh/vp5asuivqpiir5w/AAC3k_gELrP4ydNNok_o1vlYa?dl=0))

Download it and overwrite ```data/KG_VQA/fvqa/model_save```

### [Parameter](#content)
```
[--KGE {TransE,ComplEx,TransR,DistMult}] [--KGE_init KGE_INIT] [--GAE_init GAE_INIT] [--ZSL ZSL] [--entity_num {all,4302}] [--data_choice {0,1,2,3,4}]
               [--name NAME] [--no-tensorboard] --exp_name EXP_NAME [--dump_path DUMP_PATH] [--exp_id EXP_ID] [--random_seed RANDOM_SEED] [--freeze_w2v {0,1}]
               [--ans_net_lay {0,1,2}] [--fact_map {0,1}] [--relation_map {0,1}] [--now_test {0,1}] [--save_model {0,1}] [--joint_test_way {0,1}] [--top_rel TOP_REL]
               [--top_fact TOP_FACT] [--soft_score SOFT_SCORE] [--mrr MRR]
```

Available model for training: ```Up-Down```, `BAN`, `SAN`, `MLP`

**You can try your own model via adding it (`.py`) to :** `main/code/model/.`

For more details: ```code/config.py```

---

### Running
```cd code```

**For data check:**

- ```python deal_data.py --exp_name data_check```

**General VQA:**
- train:
```bash run_FVQA_train.sh```
- test:
```bash run_FVQA.sh```

**ZSL/GZSL VQA:**
- train:
```bash run_ZSL_train.sh```
- test:
```bash run_ZSL.sh```

**Note**: 
- you can open the `.sh` file for <a href="#Parameter">parameter</a> modification.

**Result:**
- Log file will be saved to: ```code/dump```
- model will be saved to: ```data/KG_VQA/fvqa/model_save```

<br />

## Explainable

![explainable](https://github.com/China-UK-ZSL/ZS-F-VQA/blob/main/figures/all_explainable.png)

<br />

## Acknowledgements
Thanks for the following released works:
>[SciencePlots](https://github.com/garrettj403/SciencePlots), [ramen](https://github.com/erobic/ramen), [GAE](https://github.com/zfjsail/gae-pytorch), [vqa-winner-cvprw-2017](https://github.com/markdtw/vqa-winner-cvprw-2017), [faster-rcnn](https://github.com/jwyang/faster-rcnn.pytorch), [VQA](https://github.com/Shivanshu-Gupta/Visual-Question-Answering), [BAN](https://github.com/jnhwkim/ban-vqa), [commonsense-kg-completion](https://github.com/allenai/commonsense-kg-completion), [bottom-up-attention-vqa](https://github.com/hengyuan-hu/bottom-up-attention-vqa), [FVQA](https://github.com/wangpengnorman/FVQA), [answer_embedding](https://github.com/hexiang-hu/answer_embedding), [torchlight](https://github.com/RamonYeung/torchlight)

## Cite:
Please condiser citing this paper if you use the code
```bigquery
@inproceedings{chen2021zero,
  title={Zero-Shot Visual Question Answering Using Knowledge Graph},
  author={Chen, Zhuo and Chen, Jiaoyan and Geng, Yuxia and Pan, Jeff Z and Yuan, Zonggang and Chen, Huajun},
  booktitle={International Semantic Web Conference},
  pages={146--162},
  year={2021},
  organization={Springer}
}
```

For more details, please submit a issue or contact [Zhuo Chen](https://github.com/hackerchenzhuo).
<a href="https://info.flagcounter.com/VOlE"><img src="https://s11.flagcounter.com/count2/VOlE/bg_FFFFFF/txt_000000/border_F7F7F7/columns_6/maxflags_12/viewers_3/labels_0/pageviews_0/flags_0/percent_0/" alt="Flag Counter" border="0"></a>