Evade-GPT-Detector

KO | EN

gitlite — search

#react #go #android #machine-learning #rust #deep-learning #docker #nodejs #macos #windows #awesome #vue #pytorch #openai

Evade-GPT-Detector

(★ 70)

Official repo for paper "Large Language Models can be Guided to Evade AI-Generated Text Detection" in TMLR 2024.

File Explorer

Download Latest Version (.zip)

- - eval.tsv
  - incontext.tsv
  - test.tsv
- - eval.tsv
  - incontext.tsv
  - test.tsv
- - eval.tsv
  - incontext.tsv
  - test.tsv
- __init__.py
- data_utils.py
- model_path.py
- my_dataloader.py
- my_logger.py
- test_utils.py
- text_utils.py
- __init__.py
- cand_generator.py
- context_optimizer.py
- incontext_trainer.py
- LLM_api.py
- prompt_constructor.py
.gitignore
cmds.sh
detectors.py
environment.yml
README.md
run_test_detection.py
shared_dir.py
SICO_test_gen.py
SICO_train.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Large Language Models can be Guided to Evade AI-Generated Text Detection

Source code for paper [*Large Language Models can be Guided to Evade AI-Generated Text Detection*](https://arxiv.org/abs/2305.10847).

## Introduction

We introduce **SICO**, a **S**ubstitution-based **I**n-**C**ontext example **O**ptimization method, 
which automatically build prompt that guide Large Language Models (LLMs), such as ChatGPT, to generate human-like texts.

SICO successfully evades all exisiting AI-generation text detectors, including GPTzero and OpenAI official detector.

## Run SICO

### Requirements
Create environment by conda:

``
conda env create -f environment.yml
``

### LLM setup

- For `ChatGPT`, you need to set OpenAI API keys in the environment `export OPENAI_API_KEY={your OpenAI API key}`.
- For `Vicuna`, please follow the instruction [here](https://github.com/lm-sys/FastChat). And set the local url in `sico/LLM_api.py`.

### Detectors setup

- `GPTzero`: Set the GPTzero API key  in environment `export GPTZERO_API_KEY={your GPTzero API key}`. Key can be obtained from [GPTzero website](https://gptzero.me/).
- `OpenAI detector`: Set the OpenAI API in environment `export OPENAI_API_KEY={your OpenAI API key}`
- `ChatGPT detector, GPT2 detector, DetectGPT, Log-Rank`: Required models will be automatically download model from HuggingFace.

### Datasets
All datasets we used are listed in `datasets` folder, containing `squad,eli5,yelp`.
Each subfolder has three tsv files: `eval.tsv` for evaluation during training, `test.tsv` for final test, and `incontext.tsv` for initialization and building in-context examples.

### Run training

Run `SICO_train.py` to start training procedure.

Here we explain each required argument in details:

- `--llm`: Base LLM used for training, including `[chatgpt, vicuna]`
- `--dataset`: Dataset we use, including `[squad, eli5, yelp]`
- `--detector`: Proxy detector we use for training, including `['chatdetect', 'gpt2detect', 'gptzero', 'openai', 'detectgpt', 'logrank']`
- `--task`: The task type of training, including `['essay', 'qa', 'rev-gen', 'paraphrase']`. Notice that `paraphrase` task matches all dataset, but `essay, qa, rev-gen` tasks only match `squad, eli5, yelp`, respectively.
- `--incontext-size`: Size of in-context examples.
- `--eval-size`: Size of evaluation data during training.
- `--train-iter`:  Maximum training iteration.

#### Examples:

Reimplement of SICO-gen on essay writing task of SQuAD dataset:

```
python SICO_train.py 
    --dataset squad 
    --llm chatgpt 
    --detector chatdetect 
    --task essay
    --incontext-size 8
    --eval-size 32
    --train-iter 6
```

Reimplement of SICO-para for open-ended question answering task:
```
python SICO_train.py 
    --dataset yelp 
    --llm chatgpt 
    --detector chatdetect 
    --task paraphrase
    --incontext-size 8
    --eval-size 32
    --train-iter 6
```

After training, the optimized prompt is stored in `./outputs/results/` and training log is stored in `./outputs/logs/`.

### Run testing

Run `SICO_test_gen.py` to use trained-prompt to generate texts, the arguments are the same as `SICO_train.py`.
Extra parameter `--test-size` show the number of cases you want to test.

After running `SICO_test_gen.py`, the generated texts are stored in `test_results/{dataset}/SICO-{dataset}-{task}-{llm}-{proxy_detector}` folder. 
Then run `run_test_detection.py` to get the AI-generated probability from different detectors, which is stored in same folder, named `{test_detector}_score.tsv`.

#### Examples:

Test SICO-gen on essay writing task of SQuAD dataset, where the prompt is trained using ChatGPT and ChatGPT Detector:
```
python SICO_test_gen.py 
    --dataset squad 
    --llm chatgpt 
    --detector chatdetect 
    --task essay
    --incontext-size 8
    --eval-size 32
    --train-iter 6
    --test-size 100
```

Test the performance of optimized prompt against `DetectGPT detector`:

```
python run_test_detection.py 
    --dataset squad
    --method SICO-squad-essay-chatgpt-chatdetect 
    --detector detectgpt
```

## To-Do List
Here is our planned roadmap for future developments:
- [x] **Open Source Code**
- [ ] **Share Effective Prompts**
- [ ] **Share Benchmark**

Stay tuned for further updates and developments on these tasks. We encourage community engagement and welcome any form of feedback or contributions to our project.

Evade-GPT-Detector

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Evade-GPT-Detector

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README