verbatim-memorization

KO | EN

gitlite — search

#javascript #python #typescript #go #machine-learning #rust #deep-learning #docker #nodejs #macos #awesome #pytorch #ios #mcp

verbatim-memorization

(★ 9)

Demystifying Verbatim Memorization in Large Language Models

verbatim-memorization 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

#	Demystifying Verbatim Memorization in Large Language Models

<div align="center">
  <a href="https://aclanthology.org/2024.emnlp-main.598/"><strong>Paper</strong></a> | <a href="https://ai.stanford.edu/blog/verbatim-memorization/"><strong>Stanford AI Lab Blog Post</strong></a> | <a href="https://nnsight.net/notebooks/mini-papers/huang_demystifying_memorization/"><strong>NNSight Mini Paper Tutorials</strong></a>
  <br>
  <br>
</div>

Verbatim memorization refers to LLMs outputting long sequences of texts that are exact matches of their training examples. In our work, we show that verbatim memorization is intertwined with the LM's general capabilities and thus will be very difficult to isolate and suppress without degrading model quality.

This repo contains:
* A framework to study verbatim memorization in a controlled setting by continuing pre-training from LLM checkpoints with injected sequences.
* Scripts using causal interventions to analyze how verbatim memorized sequences are encoded in the model representations.
* Stress testing evaluation for unlearning methods that aim to remove the verbatim memorized information.

## Data

The [data](https://github.com/explanare/verbatim-memorization/main/data) directory contains the following datasets:
* Pile data: 1M sequences sampled from the Pile, along with continuations generated by the `pythia-6.9b-deduped` model.
* Sequence injection data: 100 sequences sampled from Internet content published after Dec 2020.
* Stress testing data: 140K perturbed prefixes to evaluate whether unlearning methods truly remove the verbatim memorized information.

## Experiment

### Training with the Sequence Injection Framework

The pre-training data can be generated by the [`batch_viewer`](https://github.com/EleutherAI/pythia/blob/main/utils/batch_viewer.py) script, which allows you to extract Pythia training data between two given training steps.

The training script is at `scripts/train_with_injection.py`. For the single-shot verbatim memorization experiment, the training script is at `scripts/train_with_injection_single_shot.py`.

### Analyzing Causal Dependencies Between the Trigger and Verbatim Memorized Tokens

We use causal interventions to analyze the causal dependencies between the trigger and verbatim memorized tokens. You can find the script for causal dependency analysis on Colab:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1FX8C-Rr1tSDjklaGjMvRghcbBSzpzzBS?usp=sharing)

Below is an example of a sequence verbatim memorized by `pythia-6.9b-deduped`, which is the first sentence of the book *Harry Potter and the Philosopher's Stone*. The trigger sequence is "Mr and Mrs Dursley, of", i.e., the model can generate the full sentence given only the trigger. Yet, not all generated tokens are actually causally dependent on the trigger, e.g., the prediction of the token "you" only depends on representations of the token "thank".
 
![Causal dependencies between the trigger and verbatim memorized tokens.](https://raw.githubusercontent.com/explanare/verbatim-memorization/main/figures/causal_dependencies.svg)

### Stress Testing Unlearning Methods

The evaluation scripts, including generating perturbed prefixes, are available below:

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/19iQjGO37ifHtCM4KtH2fy99lbe2sY_vl?usp=sharing)

## Citation

If you find this repo helpful, please consider citing our work

```bibtex
@inproceedings{huang-etal-2024-demystifying,
    title = "Demystifying Verbatim Memorization in Large Language Models",
    author = "Huang, Jing  and
      Yang, Diyi  and
      Potts, Christopher",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.emnlp-main.598/",
    doi = "10.18653/v1/2024.emnlp-main.598",
    pages = "10711--10732",
}
```

verbatim-memorization

verbatim-memorization 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

verbatim-memorization

verbatim-memorization 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README