TRACE - GitLite

KO | EN

gitlite — search

#python #react #golang #android #machine-learning #rust #nodejs #macos #awesome #kubernetes #vue #awesome-list #mcp #openai

TRACE

(★ 102)

TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models

File Explorer

Download Latest Version (.zip)

- .DS_Store
- RCL.jpg
- TRACE.jpg
- eval_20Minuten.py
- eval_CStance.py
- eval_FOMC.py
- eval_MeetingBank.py
- eval_NumGLUE_cm.py
- eval_NumGLUE_ds.py
- eval_PapyrusF.py
- eval_Py150.py
- eval_ScienceQA.py
- - data_process.py
  - HHH_data_collator.py
- chatbot.py
- ICL.py
- infer_3H.py
- infer_multi.py
- infer_single.py
- - L2P.py
  - PP.py
- - __init__.py
  - EWC.py
  - GEM.py
  - LwF.py
  - O_LoRA.py
  - OGD.py
- - __init__.py
  - allnumber.pickle
  - IDBR.py
  - LFPT5.py
  - MbPAplusplus.py
- .DS_Store
- __init__.py
- base_model.py
- lora.py
- ICL.sh
- infer_3H.sh
- infer_lora.sh
- infer_seq.sh
- train_lora.sh
- train_replay.sh
- train_seq_cl.sh
- train_seq_naive.sh
- main.py
- params.py
- replay.py
- - data_cache.py
  - data_collator.py
  - data_utils.py
  - raw_datasets.py
- - bloom_flash_att.py
  - llama_flash_att.py
  - triton_flash_att.py
- - model_utils.py
- - lora.py
- - .DS_Store
  - __init__.py
  - import_utils.py
  - mapping.py
  - peft_model.py
- .DS_Store
- ds_utils.py
- utils.py
.DS_Store
.gitignore
chat.py
LICENSE
metrics.py
README.md
requirements.txt
train.py

# Installation Guide

1. Get the code

git clone https://github.com/BeyonderXX/TRACE

Downloads the entire project code from GitHub to your computer.

cd TRACE

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🌟 TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models

## 💥 Trace Benchmark

<div align=center><img src="./assets/TRACE.jpg" width="80%" /></div>

## ✨ Reasoning-Augmented Continual Learning (RCL) Method

<div align=center><img src="./assets/RCL.jpg" width="80%" /></div>

## Requirements
Our main experiments and analysis are conducted on the following environment:

- CUDA (12.2)
- torch (2.0.1)  
- torchaudio (2.0.2)  
- torchvision (0.15.2)

To install other packages, run
```
pip install -r requirements.txt
```

To use flash attention, install **flas_attn**

## data preproceess

All the data after processing can be downloaded from [`Trace Benchmark`](https://drive.google.com/file/d/1S0SmU0WEw5okW_XvP2Ns0URflNzZq6sV/view?usp=drive_link)

dataset load logic can be found in  [`utils.data.raw_datasets.py`](utils/data/data_utils.py)

1. If using datasts on huggingface，use datasets.load_dataset(ds_name) to load.
2. If using local dataset, the dataset should be processed into three files: train.json, eval.json and test.json. The format of the dataset are as follows:

```
    [{
        "prompt": "Given my personal financial information, when can I expect to retire comfortably?",
        "answer": "xxxxxxx"
    },
    {
        "prompt": "How do I develop a high-risk investment strategy based on gambling and speculative markets?",
        "answer": "xxxxxxxx"
    }]
```

data preprocess can be found in  [`utils.data.data_collator.py`](utils/data/data_collator.py)

Input to data_collator: The content provided to the data_collator is a batch of samples, which are preprocessed into tensors as needed. Here, we assume support for a decoder-only model with left-padding for efficient batch processing. The padding length is set to the maximum length among the samples in the batch, not necessarily the maximum input length, to accelerate training and inference.

## Baseline train and inference

Description of some parameters in the training and inference scripts:

- **data_path**  Path for the datasets, which in total includes nine datasets ---- eight standard training datasets (C-STANCE, FOMC, MeetingBank, Py150, ScienceQA, NumGLUE-cm, NumGLUE-ds, 20Minuten) and one Replay dataset (Lima).

- **past_task_ratio** Parameters for replay training, the replay ratio of past tasks.

- **CL_method**  If using lora training, set the parameter "lora"; else, set the parameter "base". Besides, our repository also support multi traditional continual learning methods, including "EWC", "OGD", "GEM", "MbPA++", "PP", "L2P", "LFPT5", "O-Lora".

- **inference_model_path** The folder in which the model is saved after training. Corresponding to the output_dir in the training scripts. The program will iterate through the models in the folder for inference.

**naive(full params SFT) training and inference**

```
bash scripts/train_seq_naive.sh
bash scripts/infer_seq.sh
```

**Lora training and inference**

```
bash scripts/train_lora.sh
bash scripts/infer_lora.sh
```

**Replay training and inference**

```
bash scripts/train_replay.sh
bash scripts/infer_seq.sh
```

**Continual learning methods training and inference**

```
bash scripts/train_seq_cl.sh
bash scripts/infer_seq.sh
```

**ICL**

```
bash scripts/ICL.sh
```

## Evaluation of other capabilities

### Generalization abilities evaluation

In this paper, to evaluate a model’s general ability, we assess across five key dimensions: Factual Knowledge, General Reasoning, Multilinguality, Commonsense Reasoning, and Reading Comprehension.

We follow [`Opencompass`](https://github.com/open-compass/opencompass) to evaluate the models's above abilities.

### Instruction following and safety evaluation

```
bash scripts/infer_3H.sh
```

We use GPT-4 to conduct model evluation, more details can be found in Appendix .9

## Citation
If you use our work, please cite our paper:

```latex
@article{wang2023trace,
  title={TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models},
  author={Wang, Xiao and Zhang, Yuansen and Chen, Tianze and Gao, Songyang and Jin, Senjie and Yang, Xianjun and Xi, Zhiheng and Zheng, Rui and Zou, Yicheng and Gui, Tao and others},
  journal={arXiv preprint arXiv:2310.06762},
  year={2023}
}
```