NumCLIP

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #llm #go #machine-learning #rust #nodejs #windows #kubernetes #vue #pytorch #open-source

(★ 22)

[ECCV 2024] Teach CLIP to Develop a Number Sense for Ordinal Regression

NumCLIP Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# NumCLIP

This repository contains PyTorch implementation of "Teach CLIP to Develop a Number Sense for Ordinal Regression (ECCV2024)".

Created by [Du Yao](https://scholar.google.com.hk/citations?user=8krbrWsAAAAJ&hl=zh-CN), [Zhai Qiang](https://scholar.google.com.hk/citations?hl=zh-CN&user=3I5VuhUAAAAJ), [Dai Weihang](https://scholar.google.com.hk/citations?hl=zh-CN&user=4iTfHyQAAAAJ), [Li Xiaomeng](https://xmengli.github.io/)\*

## Overview of NumCLIP

The framework of NumCLIP, aiming to teach CLIP to develop a strong number sense for ordinal regression.

![intro](https://raw.githubusercontent.com/xmed-lab/NumCLIP/main/figs/numclip.png)

## Quick Preview

### *1. Img2Lang Concept*
NumCLIP mimics human numerical cognition: mapping an image feature to a language concept first, and then reasoning the number.
<p align="center">
    <img src="figs/img_lang_num.png" width="500"> <br>

This paradigm can be condcuted in a coarse-to-fine manner. From that we elegantly convert an dense regression task into a simple and coarse classification problem, which not only smoothly mitigates the insufficient number caption issue, but also effectively utilises/recalls the pre-trained/available concept alignment learned by CLIP.

### *2. Cross-modal Ranking-based Feature Regularization*
The cross-modal negative samples are pushed away with ordinal label distance alignment.
```python
    def compute_ce_dis_loss(self,logits,y,d):

list_target = list(range(d))
        target = torch.Tensor(list_target).to('cuda:0')
        target = torch.unsqueeze(target,1)
        ls_weight = []
        for i in range(len(y)):
            label_inv_ranks = (torch.abs(y[i] - target).transpose(0,1))
            label_inv_ranks_norm = (torch.abs(y[i] - target).transpose(0,1)) / torch.sum(label_inv_ranks,dim=1) * (d-1)
            label_inv_ranks_norm = torch.squeeze(label_inv_ranks_norm,0)
            label_inv_ranks_norm[y[i]] = 1.0
            ls_label_inv_ranks_norm = label_inv_ranks_norm.detach().cpu().numpy().tolist()
            ls_weight.append(ls_label_inv_ranks_norm)

weight = torch.Tensor(ls_weight).to('cuda:0')

logits_weight = logits * weight
        loss = self.ce_loss_func(logits_weight, y)

return loss
```

## Requirements
We utilize the code base of [OrdinalCLIP](https://github.com/xk-huang/OrdinalCLIP). Please follow their instructions to prepare the environment and datasets.

## Model Training
Before training the model, move `regclipssr.py` to `./ordinalclip/models/`, and `runner_ssr.py` to `./ordinalclip/runner/` accordingly.

Add `from . import regclip_ssr` in `./ordinalclip/models/__init__.py`.

And also change the path in run.py (`from ordinalclip.runner.runner_ssr import Runner`).

```bash
sh scripts/run_regclipssr.sh
```

## What's More
Check out these amazing works leveraging CLIP for number problems!

- [OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression (NeurIPS2022)](https://github.com/xk-huang/OrdinalCLIP)
- [DepthCLIP: Can Language Understand Depth? (ACM MM2022)](https://github.com/Adonis-galaxy/DepthCLIP?tab=readme-ov-file#depthclip-can-language-understand-depth)
- [CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model (CVPR2023)](https://github.com/dk-liang/CrowdCLIP)
- [L2RCLIP: Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification (NeurIPS2023)](https://github.com/raywang335/L2RCLIP)
- [Teach CLIP to Count to Ten (ICCV2023)](https://teaching-clip-to-count.github.io/)

## Citation
If you find this codebase helpful, please consider to cite:

```
@inproceedings{du2024teach,
  title={Teach clip to develop a number sense for ordinal regression},
  author={Du, Yao and Zhai, Qiang and Dai, Weihang and Li, Xiaomeng},
  booktitle={European Conference on Computer Vision},
  pages={1--17},
  year={2024},
  organization={Springer}
}
```

NumCLIP

NumCLIP Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

NumCLIP

NumCLIP Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README