GPTAQ - GitLite

KO | EN

gitlite — search

#dotnet #proxy #cross-platform #linux #react-native #cpp #typescript #ruby #security #android

GPTAQ

(★ 94)

Code implementation of GPTAQ (https://arxiv.org/abs/2504.02692)

파일 탐색기

ZIP 다운로드

- data_utils.py
- eval_utils.py
- gptaq_utils.py
- gptq_utils.py
- hadamard_utils.py
- main.py
- model_utils.py
- monkeypatch.py
- quant_utils.py
- README.md
- requirements.txt
- rotation_utils.py
- run_llama.sh
- utils.py
- readme_intro.png
- - gptaq_utils.py
  - gptq_utils.py
  - main.py
  - modeling_llama.py
  - rotation_utils.py
- - 10_optimize_rotation.sh
  - 11_optimize_rotation_fsdp.sh
  - 2_eval_ptq.sh
  - 31_optimize_rotation_executorch.sh
  - 32_eval_ptq_executorch.sh
- - apply_r3_r4.py
  - fsdp_trainer.py
  - main.py
  - modeling_llama_quant.py
  - optimizer.py
  - quant_linear.py
  - rtn_utils.py
- - convert_to_executorch.py
  - data_utils.py
  - eval_utils.py
  - fuse_norm_utils.py
  - hadamard_utils.py
  - model_utils.py
  - monkeypatch.py
  - process_args.py
  - quant_utils.py
  - utils.py
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- LICENSE
- optimize_rotation.py
- ptq.py
- README.md
- requirement.txt
- SpinQuant.png
- data_utils.py
- eval_utils.py
- gptaq_utils.py
- gptq_utils.py
- main.py
- model_utils.py
- quant_utils.py
- README.md
- run.sh
- utils.py
.gitignore
README.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

명령어 용어집

이 문서에서 사용된 명령어를 모아봤습니다. 낯선 명령어가 있다면 펼쳐서 확인해보세요.

🔍

`pip install`

설명 보기 ▼

Install Python packages.

pip install {{package1 package2 ...}}

Install one or more packages:

pip install {{package1 package2 ...}} {{[-U|--upgrade]}}

Upgrade all specified packages to the latest version, installing any that are not already present:

pip install {{package}}=={{version}}

Install a specific version of a package:

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">  
    <p> GPTAQ: Efficient Finetuning-Free Quantization with Asymmetric Calibration [ICML 2025]</p>
</h1>  
  
<h1 align="center">   
    <img src="./img/readme_intro.png" width="1000">  
</h1>  
  
The official pytorch implementation of GPTAQ.   
  
Unlike the previous GPTQ method, which independently calibrates each layer, we always match the quantized layer’s output to the exact output in the full-precision model, resulting in a scheme that we call asymmetric calibration. Such a scheme can effectively reduce the quantization error accumulated in previous layers. We analyze this problem using optimal brain compression to derive a close-formed solution. The new solution explicitly minimizes the quantization error as well as the accumulated asymmetry error. Furthermore, we utilize various techniques to parallelize the solution calculation, including channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion. As a result, GPTAQ is easy to implement, simply using 20 more lines of code than GPTQ but improving its performance under low-bit quantization.  
  
## Update: Name change to GPTAQ

We are updating our code to the new name *`GPTAQ`*

## Update: GPTQv2 is integrated into GPTQModel

The GPTQv2 method is integrated into [GPTQModel](https://github.com/ModelCloud/GPTQModel/tree/main) library, with a simple argument to perform.

You can install GPTQModel:

```shell
pip install -v gptqmodel --no-build-isolation
```

Quantize LLaMA3.1-8B-Instruct

```python
import tempfile

from datasets import load_dataset
from gptqmodel import GPTQModel, QuantizeConfig
from gptqmodel.quantization import FORMAT
from gptqmodel.utils.eval import EVAL
from logbar import LogBar

log = LogBar.shared()

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
CFG_BITS = 4
CFG_GROUPSIZE = 128
CFG_V2 = True
INPUTS_MAX_LENGTH = 2048 # in tokens
QUANT_SAVE_PATH = f"/your_path/gptq_v2_{CFG_V2}_bit_{CFG_BITS}_gpsize_{CFG_GROUPSIZE}_llama_3.1_8B_Instruct"

def get_calib_data(tokenizer, rows: int):

calibration_dataset = load_dataset(
        "json",
        data_files="/your_path/dataset/c4-train.00000-of-01024.json.gz",
        split="train")

datas = []
    for index, sample in enumerate(calibration_dataset):
        tokenized = tokenizer(sample["text"])
        if len(tokenized.data['input_ids']) <= INPUTS_MAX_LENGTH:
            datas.append(tokenized)
            if len(datas) >= rows:
                break

return datas

quant_config = QuantizeConfig(
    bits=CFG_BITS,
    group_size=CFG_GROUPSIZE,
    format=FORMAT.GPTQ,
    desc_act=True,
    sym=True,
    v2=CFG_V2,
)

log.info(f"QuantConfig: {quant_config}")
log.info(f"Save Path: {QUANT_SAVE_PATH}")

model = GPTQModel.load(MODEL_ID, quant_config)

# load calibration data
calibration_dataset = get_calib_data(tokenizer=model.tokenizer, rows=256)

model.quantize(calibration_dataset, batch_size=1)

model.save(QUANT_SAVE_PATH)
log.info(f"Quant Model Saved to: {QUANT_SAVE_PATH}")
```

Evaluation on Arc_challenge and GSM8K:

```python
# eval
from lm_eval.tasks import TaskManager
from lm_eval.utils import make_table

with tempfile.TemporaryDirectory() as tmp_dir:
    results = GPTQModel.eval(
        QUANT_SAVE_PATH,
        tasks=[EVAL.LM_EVAL.ARC_CHALLENGE, EVAL.LM_EVAL.GSM8K_PLATINUM_COT],
        apply_chat_template=True,
        random_seed=898,
        output_path= tmp_dir,
    )

print(make_table(results))
    if "groups" in results:
        print(make_table(results, "groups"))
```

Performance comparison (GPTQv2 outperforms GPTQ on GSM8K using 1 fewer bit):

v1 ([checkpoints](https://huggingface.co/ModelCloud/GPTQ-v1-Llama-3.1-8B-Instruct
)):

|      Tasks       |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|arc_challenge|      1|none  |     0|acc     |↑  |0.5000|±  |0.0146|
|             |       |none  |     0|acc_norm|↑  |0.5128|±  |0.0146|
|gsm8k_platinum_cot|      3|flexible-extract|     8|exact_match|↑  |0.3995|±  |0.0141|
|                  |       |strict-match    |     8|exact_match|↑  |0.2548|±  |0.0125|

v2 ([checkpoints](https://huggingface.co/ModelCloud/GPTQ-v2-Llama-3.1-8B-Instruct)):

|      Tasks       |Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|------------------|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|arc_challenge|      1|none  |     0|acc     |↑  |0.5034|±  |0.0146|
|             |       |none  |     0|acc_norm|↑  |0.5068|±  |0.0146|
|gsm8k_platinum_cot|      3|flexible-extract|     8|exact_match|↑  |0.7601|±  |0.0123|
|                  |       |strict-match    |     8|exact_match|↑  |0.5211|±  |0.0144|

## Code Structure  
  
We provide several directories to reproduce the paper results.   
  
1. [**fake_quant**](./fake_quant) for reproducing QuaRot+GPTQ/GPTAQ   
2. [**spinquant**](./spinquant) for reproducing SpinQuant+GPTQ/GPTAQ  
3. [**vit_quant**](./vit_quant) for reproducing vision transformer quantization results

[//]: # (4. **GPTQModel**, a forked version of GPTQModel to support GPTQv2 to deploy weight-only quantization model  )
  
We recommend use separate envrionments for different experiments to ensure results are matched.

## Acknowledgement

Our code is built upon several repository:

[https://github.com/IST-DASLab/gptq](https://github.com/IST-DASLab/gptq)

[https://github.com/spcl/QuaRot](https://github.com/spcl/QuaRot)

[https://github.com/facebookresearch/SpinQuant/tree/main](https://github.com/facebookresearch/SpinQuant/tree/main)

## Star Histroy
[![Star History Chart](https://api.star-history.com/svg?repos=Intelligent-Computing-Lab-Panda/GPTAQ&type=Date)](https://star-history.com/#Intelligent-Computing-Lab-Yale/GPTAQ)

## Contact

Yuhang Li (*yuhang.li@yale.edu*)

## Citations  
  
If you find our work useful, please consider giving a star and citation:  
```bibtex  
@article{li2025gptqv2,
      title={GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration}, 
      author={Yuhang Li and Ruokai Yin and Donghyun Lee and Shiting Xiao and Priyadarshini Panda},
      year={2025},
      journal={arXiv preprint arXiv:2504.02692},
}
```