PDF - GitLite

KO | EN

gitlite — search

#hacktoberfest #llm #rust #deep-learning #linux #macos #java #windows #kubernetes #vue #pytorch #security #ios #openai

PDF

(★ 72)

[ICML 2024] Official implementation for "Predictive Dynamic Fusion."

File Explorer

Download Latest Version (.zip)

- - dev.jsonl
  - test.jsonl
  - test_hard_gt.jsonl
  - train.jsonl
- - dev.jsonl
  - labelResultAll.txt
  - test.jsonl
  - train.jsonl
  - val.jsonl
- batch_eval_bert.sh
- batch_eval_bow.sh
- batch_eval_concatbert.sh
- batch_eval_concatbow.sh
- batch_eval_img.sh
- batch_eval_latefusion.sh
- batch_eval_latefusion_pdf.sh
- batch_eval_mmbt.sh
- batch_train_bert.sh
- batch_train_bow.sh
- batch_train_concatbert.sh
- batch_train_concatbow.sh
- batch_train_img.sh
- batch_train_latefusion.sh
- batch_train_latefusion_pdf.sh
- batch_train_mmbt.sh
- - __init__.cpython-310.pyc
  - __init__.cpython-37.pyc
- - __init__.py
  - dataset.py
  - helpers.py
  - vocab.py
- - __init__.py
  - bert.py
  - bow.py
  - concat_bert.py
  - concat_bow.py
  - image.py
  - latefusion_pdf.py
  - mmbt.py
- - __init__.py
  - logger.py
  - utils.py
- __init__.py
frame.png
LICENSE
README.md
requirements.txt
test_pdf.py
train_pdf.py

# Installation Guide

1. Get the code

git clone https://github.com/Yinan-Xia/PDF

Downloads the entire project code from GitHub to your computer.

cd PDF

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Predictive-Dynamic-Fusion
This is the official implementation for [Predictive Dynamic Fusion](https://arxiv.org/pdf/2406.04802) (ICML 2024) by Bing Cao, Yinan Xia, Yi Ding, Changqing Zhang, and Qinghua Hu.

<p align="center">
<img src="https://github.com/YinanXia2023/Predictive-Dynamic-Fusion/blob/main/frame.png" width="850" height="500">
</p>

## Abstract
Multimodal fusion is crucial in joint decision-making systems for rendering holistic judgments. Since multimodal data changes in open environments, dynamic fusion has emerged and achieved remarkable progress in numerous applications. However, most existing dynamic multimodal fusion methods lack theoretical guarantees and easily fall into suboptimal problems, yielding unreliability and instability. To address this issue, we propose a predictive dynamic fusion (PDF) framework for multimodal learning. We proceed to reveal the multimodal fusion from a generalization perspective and theoretically derive the predictable Collaborative Belief (Co-Belief) with Mono- and Holo-Confidence, which provably reduces the upper bound of generalization error. Accordingly, we further propose a relative regularization strategy to calibrate the predicted Co-Belief for potential uncertainty. Extensive experiments on multiple benchmarks confirm our superiority.

## Environment Installation
```
numpy==1.21.6
Pillow==9.4.0
pytorch_pretrained_bert==0.6.2
scikit_learn==1.0.2
torch==1.11.0+cu113
torchvision==0.12.0+cu113
tqdm==4.65.0
```
## Dataset preparation

Step 1: Download [food101](https://www.kaggle.com/datasets/gianmarco96/upmcfood101) and [MVSA_Single](https://www.kaggle.com/datasets/vincemarcs/mvsasingle) and put them in the folder *datasets*.

Step 2: Prepare the train/dev/test splits jsonl files. We follow the [QMF](https://github.com/QingyangZhang/QMF) settings and provide them in corresponding folders.

Step 3 (optional): If you want use Glove model for Bow model, you can download [glove.840B.300d.txt](https://www.kaggle.com/datasets/takuok/glove840b300dtxt) and put it in the folder *datasets/glove_embeds*. For bert model, you can download [bert-base-uncased](https://huggingface.co/google-bert/bert-base-uncased) and put in the root folder *bert-base-uncased/*.

## Train
```
bash ./shells/batch_train_latefusion_pdf.sh
```
Tips: at the beginning of training, the output value of the confidence predictor may be minimal when batch size is small, and taking the log may be nan, which can be solved by reducing the learning rate or increasing the weight decay.

## Test
```
bash ./shells/batch_test_latefusion_pdf.sh
```

## Citation
```
@article{cao2024predictive,
  title={Predictive Dynamic Fusion},
  author={Cao, Bing and Xia, Yinan and Ding, Yi and Zhang, Changqing and Hu, Qinghua},
  journal={arXiv preprint arXiv:2406.04802},
  year={2024}
}
```

## Acknowledgement
The code is inspired by [Provable Dynamic Fusion for Low-Quality Multimodal Data](https://github.com/QingyangZhang/QMF).