Uni-Modal-Distribution-Shift

KO | EN

gitlite — search

#python #hacktoberfest #react #ai #llm #golang #deep-learning #linux #kubernetes #vue #security #ios #mcp #openai

Uni-Modal-Distribution-Shift

(★ 12)

Code for our ICML 25 paper "Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data"

Uni-Modal-Distribution-Shift 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Test-Time Selective Adaptation for Uni-Modal Distribution Shift </h1>

Uni-modal distribution shift happens when the distribution shift influences only one modality, leaving the others unchanged.

To encounter, we propose a selective adaptation schema that incorporates multiple modality-specific adaptors to accommodate potential shifts and a router module that determines which modality requires adaptation.

## Model architecture:
<div style="text-align: center;">
<img width="300" src="pic/model.jpg">
</div>

### Architecture of Pre-trained Model
For simplicity, we will illustrate our approach using two modalities: audio and video. This framework can be easily extended to more general multi-modal cases.
Suppose there is a model pre-trained on a labeled multi-modal source dataset. The model comprises modality-specific encoders (visual encoder and audio encoder), a cross-modal fusion layer, and a classifier. Both and are transformer encoders that map the input modality into a series of tokens.
The cross-modal fusion module is mainly a self-attention layer.

### Selective Adaptation Module
Given that an unknown modality would be going through shift and harm multi-modal fusion, we propose to flexibly adapt the distribution shift of different modalities.  We introduce two lightweight components on top of the base model: (I). Model-specific adapters: For the two modalities, we use two learnable matrices, $\Phi^{(v)}$ and $\Phi^{(a)}$, to fit the possible domain shift.  (II). Router: Router is parameterized by modality shift semaphore $S=[s^{(v)},s^{(a)}]$: A vector whose length equals the number of modalities, indicating the probabilities of a modality shift. In our case, the vector size is 2.  During test-time, the two components for selective adaptation are updated end-to-end.

# Data Preparation

We provide two benchmarks upon the VGGSound and Kinetics datasets for the challenge following the dataset construction as [READ (ICLR 24)](https://github.com/XLearning-SCU/2024-ICLR-READ): 
>
>You can download the benchmarks via [Google Cloud](https://drive.google.com/drive/folders/1SWkNwTqI08xbNJgz-YU2TwWHPn5Q4z5b?usp=sharing) or [Baidu Clound](https://pan.baidu.com/s/1Xo3IxQyd_fkzMVofDWKYVw?pwd=fnha) and use them according to the following steps. Notably, to make it easier, it is recommended to only modify the default root_path.
>
>
>**Step 1. Introduce corruptions for either video or audio modality**
>
>Modify the ```path``` and ```save-path```, and specify the ```corruption``` and ```severity``` to introduce the corruptions with different severity levels.
>
>
>### Video-corruption:
>python ./make_corruptions/make_c_video.py --corruption 'gaussian_noise' --severity 5 --data-path 'data_path/Kinetics50/image_mulframe_val256_k=50' --save-path 'data_path/Kinetics50/image_mulframe_val256_k-C'
>
>
>### Audio-corruption:
>python ./make_corruptions/make_c_audio.py --corruption 'gaussian_noise' --severity 5 --data_path 'data_path/Kinetics50/audio_val256_k=50' --save-path 'data_path/Kinetics50/audio_val256_k=50-C' --weather_path 'data_path/weather_audios/'
>
>
>**Step 2. Create JSON files for benchmarks**
>JSON files are needed to reproduce our code. To make it easier, we provide our JSON files as references and you can custom your own JSON files by using the following scripts.
>
>
>### JSON file for non-corrupted data (**Mandatory**):
>python ./data_process/create_clean_json.py --refer-path 'code_path/json_csv_files/ks50_test_refer.json' --video-path 'data_path/Kinetics50/image_mulframe_val256_k=50' --audio-path 'data_path/Kinetics50/audio_val256_k=50' --save-path 'code_path/json_csv_files/ks50'
>
>
>### JSON file for video-corrupted data:
>python ./data_process/create_video_c_json.py --clean-path 'code_path/json_csv_files/ks50/clean/severity_0.json' --video-c-path 'data_path/Kinetics50/image_mulframe_val256_k=50-C' --audio-path 'data_path/Kinetics50/audio_val256_k=50' --corruption 'gaussian_noise'
>
>
>### JSON file for audio-corrupted data:
>python ./data_process/create_audio_c_json.py --clean-path 'code_path/json_csv_files/ks50/clean/severity_0.json' --video-path 'data_path/Kinetics50/image_mulframe_val256_k=50' --audio-c-path 'data_path/Kinetics50/audio_val256_k=50-C' --corruption 'gaussian_noise'

# Reproduction of the main results

```
python3 run_tsa.py --dataset 'ks50' --tta-method 'TSA' --severity-start 5 --severity-end 5 --corruption-modality 'video' --gpu 0 --TSA --loss_coeff 0.5 --conf_thre 0.9 --batch-size 16
```

```
python3 run_tsa.py --dataset 'ks50' --tta-method 'TSA' --severity-start 5 --severity-end 5 --corruption-modality 'audio' --gpu 0 --TSA --loss_coeff 0.5 --conf_thre 0.9 --batch-size 64
```

```
python3 run_tsa.py --dataset 'vggsound' --json-root ./json_csv_files/VGGSound --label-csv ./json_csv_files/class_labels_indices_vgg.csv --tta-method 'TSA' --severity-start 5 --severity-end 5 --pretrain_path ./pretrained_model/vgg_65.5.pth --corruption-modality 'video' --TSA --warmup 0 --gpu 0  --loss_coeff 1 --conf_thre 0.8 --batch-size 128
```

```
python3 run_tsa.py --dataset 'vggsound' --json-root ./json_csv_files/VGGSound --label-csv ./json_csv_files/class_labels_indices_vgg.csv --tta-method 'TSA' --severity-start 5 --severity-end 5 --pretrain_path ./pretrained_model/vgg_65.5.pth --corruption-modality 'audio' --TSA --warmup 0 --gpu 0 --loss_coeff 1 --conf_thre 0.8 
```

## Main Requirement

```
numpy==1.19.2
python==3.6.13
pytorch==1.7.1 
scikit-learn==0.24.2
torchvision==0.8.2 
timm==0.9.2
```

## Citation

If our work is useful for your research, please cite the following paper:

```
@inproceedings{chen2025testtime,
    title={Test-Time Selective Adaptation for Uni-Modal Distribution Shift in Multi-Modal Data},
    author={Mingcai Chen and Baoming Zhang and Zongbo Han and Yuntao Du and Wenyu Jiang and Yanmeng Wang and Shuai Feng and Bingkun Bao},
    booktitle={Test-Time Selective Adaptation for Uni-Modal Distribution Shift},
    year={2025},
}
```

## Acknowledgements
Our project referenced the code of the following repositories. We sincerely thanks to offering useful public code base.

[READ (ICLR 24)](https://github.com/XLearning-SCU/2024-ICLR-READ)

Uni-Modal-Distribution-Shift

Uni-Modal-Distribution-Shift 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Uni-Modal-Distribution-Shift

Uni-Modal-Distribution-Shift 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README