IMF

KO | EN

gitlite — search

#python #llm #android #machine-learning #deep-learning #linux #docker #java #windows #awesome #cli #vue #awesome-list #ios

(★ 79)

[IEEE TCSVT2024] Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration

File Explorer

Download Latest Version (.zip)

- - profiles_settings.xml
- .gitignore
- IMF.iml
- misc.xml
- modules.xml
- vcs.xml
- workspace.xml
- fuse_Ycbcr_rgb.py
- generate_affine_deform_data.py
- get_svm_map_softmax.py
- fuse_data_vsm.py
- joint_data.py
- reg_data.py
- metrics.py
- Fusion.png
- MPDR-TCF.png
- network.png
- Reg.png
- Visual_Fus.png
- Visual_Reg.png
- affine_transform.py
- elastic_transform.py
- fusion_SA_loss.py
- ms_ssim.py
- reg_losses.py
- ssim_attention_loss.py
- ssimloss.py
- TV_loss.py
- co_fusion_net.py
- mpdrnet.py
- test_co_fuse.py
- test_co_fuse_ycbcr.py
- test_reg.py
- test_reg_co_fusion.py
- train_affi_defo_reg.py
- train_co_fuse.py
- train_reg.py
- train_reg_co_fusion_sa.py
.gitattributes
img.png
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# IMF
 
[![LICENSE](https://img.shields.io/badge/license-MIT-green)](https://github.com/wdhudiekou/IMF/blob/main/LICENSE)
[![Python](https://img.shields.io/badge/python-3.6-blue.svg)](https://www.python.org/)
[![PyTorch](https://img.shields.io/badge/pytorch-1.6.0-%237732a8)](https://pytorch.org/)

### Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration [IEEE TCSVT2024]

By Di Wang, Jinyuan Liu, Long Ma, Risheng Liu, and Xin Fan*

## Updates  
[2024-06-08] Our paper is accepted to IEEE TCSVT !

[2023-08-25] Our paper is available online! [[arXiv version](https://arxiv.org/pdf/2308.11165.pdf)]

## Requirements
- CUDA 10.1
- Python 3.6 (or later)
- Pytorch 1.6.0
- Torchvision 0.7.0
- OpenCV 3.4
- Kornia 0.5.11

## Data preparation
1. You can generate misaligned infrared-visible images for training/testing by
    ```python
       cd ./data
       python generate_affine_deform_data.py
   
In 'Trainer/train_reg.py', deformable infrared images are generated in real time by default during training.

2. You can obtain self-visual saliency maps for training the fusion process of infrared and visible images by
    ```python
       cd ./data
       python get_svs_map_softmax.py
   
 ## Get start
1. You can use the pseudo infrared images [[link](https://pan.baidu.com/s/1M79RuHVe6udKhcJIA7yXgA) code: qqyj] generated by the CPSTN proposed by [UMF](https://github.com/wdhudiekou/UMF-CMGR) to train/test our C-MPDR:
    ```python
       cd ./Trainer
       python train_reg.py

cd ./Test
       python test_reg.py
  
2. If you tend to train Registration and Fusion subnetworks separately, You can run following commands:

```python
       cd ./Trainer
       python train_reg.py

cd ./Trainer
       python train_co_fuse.py
  The corresponding test code 'test_reg.py' and 'test_co_fuse.py' can be found in 'Test' folder.

3. If you tend to train Registration and Fusion subnetworks jointly, You can run following command: 
   ```python
       cd ./Trainer
       python train_reg_co_fusion_sa.py

The corresponding test code 'test_reg_co_fusion.py' can be found in 'Test' folder.

## Dataset
Please download the following datasets:
*   [RoadScene](https://github.com/hanna-xu/RoadScene)
*   [TNO](http://figshare.com/articles/TNO\_Image\_Fusion\_Dataset/1008029)
*   [M3FD](https://github.com/JinyuanLiu-CV/TarDAL)
*   [MSIFT](https://ivrlwww.epfl.ch/supplementary_material/cvpr11/index.html)

<b>Note: The above datasets are manually pre-registered. Desired misaligned image can be generated using the proposed image synthesis method.</b>

## Pretrained Models
1. Pretrained models of registration subnetwork <b> MPDR </b> are as follows:
*   [RoadScene](https://pan.baidu.com/s/1HIqDsxBJFVASRXfTrdcp-w) (code: wyi0) or [RoadScene](https://drive.google.com/file/d/1XyvUyWTiyOwcWCxAPg7FnZHeeDAIrT1f/view?usp=drive_link) (Google Link)
*   [TNO](https://pan.baidu.com/s/1wBfJec-ryY2h2bmjxmTrfg) (code: qul9) or [TNO](https://drive.google.com/file/d/1Hh8909sCjihqcFE9wo11R5sUE72ycCH7/view?usp=drive_link) (Google Link)
*   [M3FD](https://pan.baidu.com/s/1OUJIYyH5SEY-RO5VU-beQg) (code: uq0b) or [M3FD](https://drive.google.com/file/d/1NOHXhg_sXocEV7IP-qAgrQT4NtjBe9m3/view?usp=drive_link) (Google Link)
*   [MSIFT](https://pan.baidu.com/s/1GVohJKukbtck_Tvzqi8K_Q) (code: 410b) or [MSIFT](https://drive.google.com/file/d/1NuKUuc9_-oqJjp8kcJ27YHhJRoUXSZyY/view?usp=drive_link) (Google Link)

2. Pretrained models of fusion subnetwork <b> TCF </b> are as follows:
*   [RoadScene](https://pan.baidu.com/s/1KqJJhYDYqHTE-dP8UlGPyg) (code: nju6) or [RoadScene](https://drive.google.com/file/d/1wHCusEuui8BS6gz8D2Hf788fl81pyQjz/view?usp=drive_link) (Google Link)
*   [TNO](https://pan.baidu.com/s/1nsz2rkaJ15HAZzdjZEvEZQ) (code: rqmx) or [TNO](https://drive.google.com/file/d/1SKkF93DwydoO_SfTAwR3RkJ6_5I6oLim/view?usp=drive_link) (Google Link)
*   [M3FD](https://pan.baidu.com/s/194WV0_G5B8Y2ORlwMqD6xQ) (code: 0rd4) or [M3FD](https://drive.google.com/file/d/1A5D9RcsFYBgb8TYUiEmkteMKlFIJQ6QU/view?usp=drive_link) (Google Link)

## Experimental Results
1. Please download the <b> Registration</b> results by our <b> IMFusion </b>:
*  [RoadScene](https://pan.baidu.com/s/1ntxhecwXJhD2Yc82d5caOw ) (code: pv7s)
*  [TNO](https://pan.baidu.com/s/13LUuUkmUulbbpHbsmm60sw ) (code: yyf8)
*  [M3FD](https://pan.baidu.com/s/1djvhlCBjWHX4Q80ZKO38FA ) (code: 7812)
*  [MSIFT](https://pan.baidu.com/s/1QxqkUoD7xoHiaevnCiRwvQ ) (code: ahfs)

2. Please download the <b> Fusion </b> results by our <b> IMFusion </b>:
*  [RoadScene](https://pan.baidu.com/s/1XPUYGDWSK95NwnA9qmFCJA ) (code: 5rbw)
*  [TNO](https://pan.baidu.com/s/1n3TZjDZffr0vNOUsFU0wkw ) (code: y29p)
*  [M3FD](https://pan.baidu.com/s/1OLzvgP9UE_IMKBAZLPOFIg ) (code: 2x64)

## Related Projects

[UMF](https://github.com/wdhudiekou/UMF-CMGR) (Published in IJCAI 2022!)

## Citation
```
@article{Wang_2024_IMF,
      author={Di Wang and Jinyuan Liu and Long Ma and Risheng Liu and Xin Fan},
      title={Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration}, 
      journal={{IEEE} Transactions on Circuits and Systems for Video Technology},
      volume={34},
      number={11},
      pages={10944--10958},
      year={2024}
}
```

IMF

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

IMF

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README