STONE-watermarking

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #typescript #llm #go #deep-learning #docker #nodejs #awesome #pytorch #mcp #ios

STONE-watermarking

(★ 13)

Official repository of the paper: Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code (Findings of EACL 2026)

STONE-watermarking Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Marking Code Without Breaking It: Code Watermarking for Detecting LLM-Generated Code (Findings of EACL 2026)

[**📖 Paper**](https://arxiv.org/pdf/2502.18851)

## TL; DR
We present STONE, a syntax-aware code watermarking method that embeds watermarks only into non-syntax tokens, ensuring reliable detection of AI-generated code while preserving functional correctness. We also introduce STEM, a unified evaluation metric for assessing watermark effectiveness across correctness, detectability, and imperceptibility.

## About STONE 
**STONE (Syntax TOkeN preserving codE watermarking)** is a robust, syntax-aware watermarking approach designed specifically for code generated by large language models. It strategically embeds watermarks only in non-syntax tokens, preserving critical elements such as keywords, operators, and delimiters, to ensure that the functionality and structural integrity of the code remain intact. By modifying only tokens with minimal impact on code execution, STONE provides a reliable and efficient solution to trace the origin of generated code while maintaining optimal performance.

## About STEM
**STEM (Summative Test Metric for Evaluating code-waterMarking)** is a comprehensive metric that objectively assesses watermarking techniques across three key dimensions: correctness, detectability, and imperceptibility. It quantifies functional correctness using pass@k metrics, evaluates watermark detectability through statistical measures such as z-scores and AUROC, and ensures that the presence of a watermark does not compromise the natural token distribution of code by measuring perplexity. This balanced metric enables clear comparisons among different methods and guides the optimization of watermarking strategies to enhance transparency and accountability in code generated by large language models.

The overall performance of a watermarking technique is evaluated using STEM, which integrates functional correctness, detectability, and imperceptibility into a single metric:

$$STEM = \alpha \cdot \mathrm{Correctness}(C_{wm}) + \beta \cdot \mathrm{Detectability}(C_{wm}, C_{H}) + \zeta \cdot \mathrm{Naturalness}(C_{wm},C)$$

with the condition that $\alpha + \beta + \zeta = 1$.

The individual components are defined as follows:

### 1. Correctness

$$\mathrm{Correctness}(C_{wm}) = \mathbb{E}_{C_{wm}}\left[ 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}} \right]$$

Correctness evaluates whether the code works correctly before and after watermark insertion.
- $n$: Total number of code samples generated for a given problem
- $c$ : Number of code samples that work correctly
- $\binom{n}{k}$ : Number of ways to choose $k$ samples from $n$ samples
- $\binom{n - c}{k}$ : Number of ways to choose $k$ samples consisting only of incorrect code (i.e., excluding the correct ones)
- Therefore, $1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}$ represents the probability that at least one correct code sample is included when $k$ samples are selected, and the overall correctness score is evaluated as the expectation of this probability over the watermark code set $C_{wm}$.

### 2. Detectability
Detectability evaluates how clearly the watermark is embedded in the code using statistical metrics such as the z-score and AUROC.

#### 2-1. z-score: Green Token Ratio
$$z(X_{wm}) = \frac{|X_{wm}|_G - \gamma |X_{wm}|}{\sqrt{\gamma (1 - \gamma)|X_{wm}|}}$$

- $∣X_{wm}∣_G$ : The number of green list tokens in the watermarked code
- $|X_{wm}|$ : Total number of tokens in the code
- $\gamma$: Expected proportion of green tokens in non-watermarked code
- The denominator represents the standard deviation under a binomial distribution. 
- The z-score indicates how many standard deviations the actual count of green tokens deviates from the expected count; a larger value means the watermark is more clearly present.

#### 2-2. Detecting LLM-Generated Code Using the Green Token Ratio
To determine whether the generated code has a watermark, various threshold values $\tau$ are applied to compute the True Positive Rate (TPR) and False Positive Rate (FPR).
- True Positive Rate (TPR): For watermarked code $C_{wm}$, TPR is the proportion of code samples with a z-score exceeding the threshold $\tau$.

$$\text{TPR}(\tau) = \frac{\sum_{j}^{J} \mathbf{1}[z(X_{wm}^{(j)}) > \tau]}{J}$$

- False Positive Rate (FPR): For non-watermarked code $C_H$, FPR is the proportion of code samples with a z-score exceeding the threshold $\tau$:

$$\text{FPR}(\tau) = \frac{\sum_{i}^{I} \mathbf{1}[z(X_H^{(i)}) > \tau]}{I}$$

#### 2-3. Overall Detectability Using AUROC
$$\mathrm{Detectability}(C_{wm}, C_H) = \int_{0}^{1} \text{TPR}(\text{FPR})d(\text{FPR})$$

- The Detectability score is obtained by integrating the relationship between TPR and FPR over all threshold values $\tau$, which calculates the AUROC.
- An AUROC value close to 1 indicates that the watermark is clearly distinguishable, whereas a value near 0.5 indicates that watermarked and non-watermarked code are difficult to distinguish.
- A high AUROC value demonstrates that the distribution differences of the z-scores based on the green token ratio effectively separate watermarked code from non-watermarked code.

### 3. Imperceptibility
Imperceptibility assesses whether the code remains fluent (in terms of Perplexity) after watermark insertion, ensuring that it appears natural to human readers.

The difference in Perplexity between non-watermarked code $C_{H}$ and watermarked code $C_{wm}$ is normalized to measure imperceptibility.

```math
PPL(C_{wm})=\frac{1}{|C_{wm}|}\sum_{j=1}^{|C_{wm}|} e^{-\frac{1}{N_j}\sum_{i=1}^{N_j}\log P(y_i^{(j)} \mid y_{1:i-1}^{(j)})}
```

- $N_j$: Number of tokens in the j-th code sample in $C_{wm}$.
- $P(y_i^{(j)} | y_{<i}^{(j)})$: The probability of token $y_i^{(j)}$ given its preceding context in the j-th sample, where $y_i^{(j)}$ represents the i-th token in the j-th code sample

Then, the Imperceptibility score is defined as:

$$\mathrm{Naturalness}(C_{wm}, C) = 1 - \frac{|\text{PPL}(C_{wm}) - \text{PPL}(C)|}{\text{PPL}(C)}$$

- The Imperceptibility score normalizes the absolute difference in Perplexity between watermarked and non-watermarked code by the Perplexity of the original code C.
- A score closer to 1 indicates that the watermark has little effect on the natural flow of the code, meaning the watermark is well concealed.

## Getting Started

### Project Structure
```
STONE-watermarking/
├── stone.yaml                  # Conda environment configuration file
├── .env                        # API key file (to be created by the user)
├── stone_implementation/       # Main implementation directory
│   ├── run.sh                  # Script to generate watermarked code
│   ├── custom_evalplus/        # Custom evaluation scripts
│   │   └── evalplus/
│   │       └── pass_evaluation.sh  # Script to evaluate HumanEvalplus and MBPPplus
│   ├── evaluation/            # Evaluation scripts and notebooks
│   │   ├── AUROC_perplexity.ipynb # Notebook to calculate detectability and imperceptibility scores
│   │   └── stem.py            # Script to calculate STEM score
│   └── CodeIP/                # CodeIP implementation directory
│       └── examples/
│           └── src/
│               ├── watermarking/           # Watermarking implementation
│               │   ├── watermark_processors/  # Different watermarking methods
│               │   ├── utils/               # Utility functions
│               │   └── wm.py               # Main watermarking script
│               ├── humanevalpack/          # HumanEvalPack dataset
│               │   └── data/               # Dataset files for different languages
│               ├── output/                 # Output directory
│               │   ├── convert.py          # Script to convert output format
│               │   ├── perplexity.py       # Script to calculate perplexity
│               │   └── pass_evaluation.sh  # Script to evaluate CodeIP results
│               └── run_wm.sh              # Script to run CodeIP watermarking
└── README.md                   # Project documentation
```

### Installation

Environment settings:
```bash
git clone "[OUR REPO]"
conda env create -f stone.yaml
```
API key settings:
```bash
#make .env file and put your api key
HF_ACCESS_TOKEN="[INSERT YOUR API KEY]"
```

### Training
1-1. Train HumanEval+, MBPP+, and HumanEvalPack
Generating watermarked machined-generated code:
```bash
cd stone_implementation
bash run.sh
```

1-2. Train CodeIP
Generating watermarked machined-generated code:
```bash
cd stone_implementation/CodeIP/examples/src
bash run_wm.sh
```

### Evaluating

### 1. Calculating correctness score

1-1. Evaluate HumanEval+ and MBPP+
```bash
cd stone_implementation/custom_evalplus/evalplus
bash pass_evaluation.sh
```

1-2. Evaluating HumanEvalPack

> 1) Prepare Result Files
```bash
# Compress files needed for evaluation
tar -cvf bigcode-evaluation-harness.tar.gz bigcode-evaluation-harness
tar -cvf results.tar.gz ./stone_implementation/results
```

> 2) Google Colab Setup
1. Upload the compressed files to Google Drive
2. Run the following code in a Colab notebook:

```python
# Mount Google Drive
from google.colab import drive
drive.mount('/content/drive')

# Set up working directory and extract files
%cd /content/drive/MyDrive/
!tar -xvf results.tar.gz
!tar -xvf bigcode-evaluation-harness.tar.gz

# Install required packages
%cd /content/drive/MyDrive/bigcode-evaluation-harness
!pip install -r requirements.txt
!pip install datasets evaluate
```

> 3) Run Evaluation
```python
# Evaluation settings
language = 'cpp'      # Programming language to evaluate
model = 'qwen'        # Model to use
gamma = 0.5           # Gamma 
delta = 0.5           # Delta
method = 'STONE'      # Watermarking method
hash_key = '15485863' # Hash key
# Set input file path
input_file_path = f"/content/drive/MyDrive/results/5samples/humanevalpack_{language}_{model}_{method}_all_pl_False_{gamma}_{delta}_{hash_key}__watermarked_solutions.json"

# Run evaluation
!python main.py \
    --tasks humanevalsynthesize-{language} \
    --n_samples 5 \
    --batch_size 5 \
    --allow_code_execution \
    --postprocess \
    --trust_remote_code \
    --load_generations_path {input_file_path}
```

1-3. For CodeIP
```bash
cd stone_implementation/CodeIP/examples/src/output
python convert.py
bash pass_evaluation.sh
```

### 2. Getting detectability and imperceptibility scores

2-1. For other baseline and STONE
```bash
cd stone_implementation/evaluation
AUROC_perplexity.ipynb
```
2-2. For CodeIP
```bash
#1) Getting detectability score
cd stone_implementation/CodeIP/examples/src/output
# Check the "acc" value in the "codeip_results.json" file and see "extration_rate"

#2) Getting imperceptibility score
cd stone_implementation/CodeIP/examples/src/output
python perplexity.py
```

### 3. Calculating STEM
Combine the obtained correctness, detectability, and imperceptibility scores to calculate the STEM score.
```bash
cd stone_implementation/evaluation
stem.py
```

## 🖊 Citation
```text
@inproceedings{kim-etal-2026-marking,
    title = {Marking Code Without Breaking It: Code Watermarking for Detecting {LLM}-Generated Code},
    author = {Kim, Jungin and Park, Shinwoo and Han, Yo-Sub},
    booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
    month = mar,
    year = {2026},
    address = {Rabat, Morocco},
    publisher = {Association for Computational Linguistics},
    url = {https://aclanthology.org/2026.findings-eacl.207/},
    doi = {10.18653/v1/2026.findings-eacl.207},
    pages = {3990--4002},
    ISBN = {979-8-89176-386-9}
}
```

## Acknowledgements
This repository is based on the codes of [MarkLLM](https://github.com/THU-BPM/MarkLLM), [CodeIP](https://github.com/CGCL-codes/naturalcc/tree/main/examples/codeip) and [bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness)

STONE-watermarking

STONE-watermarking Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

STONE-watermarking

STONE-watermarking Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README