TokenFlow - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #typescript #go #golang #android #linux #docker #macos #java #cli #kubernetes #vue #ios

TokenFlow

(★ 464)

[CVPR 2025] 🔥 Official impl. of "TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation".

File Explorer

Download Latest Version (.zip)

- radar.png
- teasor.png
- - __init__.py
  - constants.py
  - conversation.py
  - mm_utils.py
  - utils.py
- cog.yaml
- pyproject.toml
- - __init__.py
  - constants.py
  - conversation.py
  - mm_utils.py
  - utils.py
- - run_t2i.sh
  - train_t2i.sh
- requirements.txt
- - augmentation.py
  - build.py
  - coco.py
  - imagenet.py
  - openimage.py
- - reconstruction_vq.sh
  - reconstruction_vq_siglip.sh
  - train_vq.sh
  - train_vq_finetune_decoder.sh
  - train_vq_finetune_enhanced_decoder.sh
- - discriminator.py
  - discriminator_patchgan.py
  - discriminator_stylegan.py
  - lpips.py
  - norm_ema_quantizer.py
  - reconstruction_vq_ddp.py
  - transforms.py
  - vq_loss.py
  - vq_model.py
  - vq_train.py
  - vqkd_model.py
- - data.py
  - deepspeed.py
  - distributed.py
  - drop_path.py
  - ema.py
  - logger.py
- requirements_tokenflow.txt
.gitignore
GETTING_STARTED.md
LICENSE
README.md

# Installation Guide

1. Get the code

git clone https://github.com/ByteVisionLab/TokenFlow

Downloads the entire project code from GitHub to your computer.

cd TokenFlow

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r t2i/requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# TokenFlow🚀: Unified Image Tokenizer for Multimodal Understanding and Generation

<div align="center">

[![TokenFlow](https://img.shields.io/badge/Paper-TokenFlow-2b9348.svg?logo=arXiv)](https://arxiv.org/abs/2412.03069)&nbsp;
[![huggingface weights](https://img.shields.io/badge/%F0%9F%A4%97%20Weights-TokenFlow-yellow)](https://huggingface.co/collections/ByteVisionLab/tokenflow-674ebd443b616dff1a634178)&nbsp;
[![project page](https://img.shields.io/badge/Project_page-More_visualizations-green?logo=bytedance)](https://bytevisionlab.github.io/TokenFlow/)&nbsp;
![Visitors](https://visitor-badge.laobi.icu/badge?page_id=ByteFlow-AI.TokenFlow)
[![GitHub closed issues](https://img.shields.io/github/issues-closed/ByteFlow-AI/TokenFlow?color=blue&label=Issues)](https://github.com/ByteFlow-AI/TokenFlow/issues?q=is%3Aissue+is%3Aclosed)

</div>

## 🌿 Introduction

We present TokenFlow, a unified image tokenizer that bridges the long-standing gap between multimodal understanding and generation. 
TokenFlow introduce an innovative dual-codebook architecture that decouples semantic and pixel-level feature learning while maintaining their alignment through a shared mapping mechanism.

<div align='center'>
<img src="./assets/radar.png" class="interpolation-image" alt="radar." width="50%" />
</div>

TokenFlow excels in both multimodal understanding and image generation. For multimodal understanding, we surpass the flagship models such as LLaVA-1.5 and EMU3 by a large margin. For text-to-image generation, we also achieve comparable performance to SDXL in 256×256 resolution.

<div align='center'>
<img src="./assets/teasor.png" class="interpolation-image" alt="teasor." width="100%" />
</div>

## 📰 News

**2025.02.27**: TokenFlow got accepted to CVPR 2025.

**2024.12.9**:  Code and checkpoints are released.

**2024.12.5**:  🎉🎉🎉 TokenFlow is released! 🎉🎉🎉  See our [project page](https://byteflow-ai.github.io/TokenFlow/) and [paper](https://arxiv.org/abs/2412.03069) .

## ⚙️ Getting Started

See [GETTING_STARTED.md](./GETTING_STARTED.md) for detailed instructions of ***training*** and ***evaluation*** of (1) TokenFlow, (2) multimodal understanding model and (3) text-to-image generation model.

## 🤗 Checkpoints

**Text-to-Image Model**

<table>
  <tr>
    <th style="width: 150px;">Model Size</th>
    <th>Tokenizer Weight</th>
    <th>Model Weight</th>
  </tr>
  <tr>
    <td align="center">7B</td>
    <td align="center"><a href="https://huggingface.co/ByteFlow-AI/TokenFlow/blob/main/tokenflow_clipb_32k_enhanced.pt">TokenFlow</a></td>
    <td align="center"><a href="https://huggingface.co/ByteFlow-AI/TokenFlow-t2i">TokenFlow-t2i</a></td>
  </tr>
</table>

**Multimodal Understanding Model**

<table>
  <tr>
    <th style="width: 150px;">Language Backbone</th>
    <th>Tokenizer Weight</th>
    <th>Model Weight</th>
  </tr>
  <tr>
    <td align="center">Qwen-2.5-14B</td>
    <td align="center"><a href="https://huggingface.co/ByteFlow-AI/TokenFlow/blob/main/tokenflow_siglip_32k.pt">TokenFlow-XL</a></td>
    <td align="center"><a href="https://huggingface.co/ByteFlow-AI/Tokenflow-llava-qwen2.5-14B-finetuning">TokenFlow-llava-qwen2.5-14B-finetuning</a></td>
  </tr>
</table>

## 📑 Open-source Plan

- [X] Release the checkpoint of tokenizer, text-to-image model & multimodal understanding model.
- [X] Release the training & inference code for tokenizer.
- [X] Release the training & inference code for text-to-image generation.
- [X] Release the training & inference code for multimodal understanding.

## Acknowledgement

We thank the great work from [VAR](https://github.com/FoundationVision/VAR), [LlamaGen](https://github.com/FoundationVision/LlamaGen) and [LLaVA](https://github.com/haotian-liu/LLaVA).

## 📄 Citation

If our work assists your research, feel free to give us a star ⭐ or cite us using

```
@article{qu2024tokenflow,
  title={Tokenflow: Unified image tokenizer for multimodal understanding and generation},
  author={Qu, Liao and Zhang, Huichao and Liu, Yiheng and Wang, Xu and Jiang, Yi and Gao, Yiming and Ye, Hu and Du, Daniel K and Yuan, Zehuan and Wu, Xinglong},
  journal={arXiv preprint arXiv:2412.03069},
  year={2024}
}
```

## 🔥 Open positions
We are hiring interns and full-time researchers at ByteDance, with a focus on multimodal understanding and generation (preferred base: Hangzhou, Beijing, and Shenzhen). If you are interested, please contact quliao1117@gmail.com.