DiJiang - GitLite

KO | EN

gitlite — search

#javascript #android #machine-learning #deep-learning #docker #macos #java #awesome #vue #awesome-list #security #pytorch #ios #mcp

DiJiang

(★ 103)

[ICML'24 Oral] The official code of "DiJiang: Efficient Large Language Models through Compact Kernelization", a novel DCT-based linear attention mechanism.

파일 탐색기

최종 버전 다운로드 (.zip)

- dijiang.png
- dijiang_speed.png
- experiments1.png
- experiments2.png
- scheme.png
- - config.json
  - modeling_gpt_neox_dijiang.py
  - special_tokens_map.json
  - tokenizer.json
  - tokenizer_config.json
- - config.json
  - modeling_gpt_neox_dijiang.py
  - special_tokens_map.json
  - tokenizer.json
  - tokenizer_config.json
- - config.json
  - modeling_gpt_neox_dijiang.py
  - special_tokens_map.json
  - tokenizer.json
  - tokenizer_config.json
- - config.json
  - modeling_gpt_neox_dijiang.py
  - special_tokens_map.json
  - tokenizer.json
  - tokenizer_config.json
- - config.json
  - modeling_gpt_neox_dijiang.py
  - special_tokens_map.json
  - tokenizer.json
  - tokenizer_config.json
README.md
requirements.txt
run_dijiang.sh
train_pythia.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/YuchuanTian/DiJiang

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd DiJiang

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# DiJiang: Efficient Large Language Models through Compact Kernelization

<p align="left">
<a href="https://arxiv.org/abs/2403.19928" alt="arXiv">
    <img src="https://img.shields.io/badge/arXiv-2403.19928-b31b1b.svg?style=flat" /></a>
</p>

## News

**6/14/2024** Our work DiJiang is accepted as **ICML'24 Oral**!🎉🎉🎉

<img src="./imgs/dijiang.png" alt="scheme" style="zoom: 20%;"/>

This is the official implementation of ["DiJiang: Efficient Large Language Models through Compact Kernelization"](https://arxiv.org/abs/2403.19928), a novel Frequency Domain Kernelization approach to linear attention. DiJiang is a mythical monster in Chinese tradition. It has six legs and four wings , and it is known for its capability to move quickly.

![scheme](https://raw.githubusercontent.com/YuchuanTian/DiJiang/main/imgs/scheme.png)

🚀 In our work "DiJiang", we propose a fast linear attention approach that leverages the power of **Fast Discrete Cosine Transform (Fast DCT).** Theoretical analyses and extensive experiments demonstrate the speed advantage of our method.

<img src="./imgs/dijiang_speed.png" alt="scheme" style="zoom: 35%;" />

<img src="./imgs/experiments1.png" alt="scheme" style="zoom: 67%;" />

<img src="./imgs/experiments2.png" alt="scheme" style="zoom: 67%;" />

## Training

### Preparation

- Download Dataset: We utilized the Pile dataset for training [The Pile: An 800GB Dataset of Diverse Text for Language Modeling (arxiv.org)](https://arxiv.org/abs/2101.00027)
- Check CUDA version: suggested cuda>=11.7

- Install required packages: ```pip install -r requirements.txt```

- Download pre=trained model weights (i.e. ```pytorch_model.bin``` files in the original HuggingFace models) and place them in the corresponding DiJiang model folder (```e.g. modeling/pythia-70M-dijiang```).

Links of pre-trained weights:

```
# Pythia-70M
https://huggingface.co/EleutherAI/pythia-70m/resolve/main/pytorch_model.bin?download=true

# Pythia-160M
https://huggingface.co/EleutherAI/pythia-160m/resolve/main/pytorch_model.bin?download=true

# Pythia-410M
https://huggingface.co/EleutherAI/pythia-410m/resolve/main/pytorch_model.bin?download=true

# Pythia-1B
https://huggingface.co/EleutherAI/pythia-1b/resolve/main/pytorch_model.bin?download=true

# Pythia-2.8B
https://huggingface.co/EleutherAI/pythia-2.8b/resolve/main/pytorch_model.bin?download=true

```

### Running

A training shell is provided at ```bash run_dijiang.sh``` for your reference.

Important arguments to take care of:

```

--model_name_or_path <path to the HuggingFace model>
    --dataset_dir <path to the original PILE dataset>
    --data_cache_dir <path to the cached dataset (if cached); cache stored at this dir>
    --per_device_train_batch_size <global_batchsize // device_count>
    --output_dir <dir to store the trained model>
    --logging_dir <dir to store tensorboard>

```

## Future work (Stay Tuned!)
- [x] Training code for **DiJiang**
- [ ] Better inference code
- [ ] Inference models

## Citation (in BibTex)

```
@misc{chen2024dijiang,
      title={DiJiang: Efficient Large Language Models through Compact Kernelization}, 
      author={Hanting Chen and Zhicheng Liu and Xutao Wang and Yuchuan Tian and Yunhe Wang},
      year={2024},
      eprint={2403.19928},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}
```

## Acknowledgement

We acknowledge the authors of the following repos:

[DenseSSM (github)](https://github.com/WailordHe/DenseSSM)

[Llama2 (huggingface.co)](https://huggingface.co/docs/transformers/main/model_doc/llama2#usage-tips)

[Pythia (EleutherAI)](https://huggingface.co/docs/transformers/main/model_doc/llama2#usage-tips)

[RetNet (Microsoft)](https://github.com/microsoft/torchscale/tree/main)

[Transformersscale (Microsoft)](https://github.com/microsoft/transformersscale/tree/main)

[Stanford Alpaca (Stanford)](https://github.com/tatsu-lab/stanford_alpaca)

[Chinese-LLaMA-Alpaca (ymcui)](https://github.com/ymcui/Chinese-LLaMA-Alpaca)