econ-ai-detector

KO | EN

gitlite — search

#cloud-native #graphql #rust #claude #macos #ui #json #css #cpp #cross-platform

econ-ai-detector

(★ 9)

Open detector of AI-generated text in economics working papers (TF-IDF LR + DistilRoBERTa ensemble from the pangram-nber project)

File Explorer

ZIP Download

- __init__.py
- cli.py
- detector.py
- preprocess.py
- sentseg.py
- lr_v3.joblib
- thresholds.json
.gitignore
LICENSE
pyproject.toml
README.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

An open, from-scratch detector of AI-generated text in economics working papers.
It is the "toy" detector built alongside the [pangram-nber](https://github.com/paulgp/pangram-nber)
project, which measures AI-written text in NBER working papers and arXiv economics
preprints with a commercial detector (Pangram). This repository releases the trained
v3 ensemble so anyone can score their own papers without a commercial API.

## What it is

Two window-level scorers and one paper-level rule.

| Component | What it is | Output |
|---|---|---|
| LR | word (1–2 gram) + character (3–5 gram) TF-IDF logistic regression, `models/lr_v3.joblib` | P(AI) per 250-word window |
| NN | DistilRoBERTa-base fine-tuned classifier, hosted on Hugging Face as `paulgp85/econ-ai-detector-distilroberta-v3` | logit margin (AI − human) per window |
| Rule | paper flagged if its **2nd-highest** LR probability ≥ 0.335 **and** its 2nd-highest NN margin ≥ 7.48 | flagged / not flagged |

The k = 2 rule exploits a regularity in the calibration data: human false positives are
isolated windows, while genuinely AI-written papers flag several. Thresholds for three
operating points (paper-level false-positive rate 0.1%, 0.5%, 1%) are in
`models/thresholds.json`; 0.1% is the default.

**Training data.** Human text: ~7,000 pre-2020 NBER working papers and 2020–24 published
journal articles (prose-gated windows). AI text: ~5,000 "mirror" passages generated by
Qwen3.6-35B as topic-matched continuations of those same papers, plus adversarial mirrors
and mined hard negatives from three rounds of Pangram-style reinforcement. No 2022 or
later NBER paper was used in training, so the 2022 NBER census is a clean placebo.

**Operating characteristics at the default 0.1% point** (paper-level, intro + one body
section scored, as in the census):

| Register | Papers | False-positive rate |
|---|---|---|
| Pre-2020 NBER/journal test shard (held out) | ~1,370 | 0.00% |
| Published journal articles 2020–24 | ~5,300 | 0.06% |
| NBER working papers, 2022 (never trained on) | 599 | 0.17% |

At that point it flags 13.4% of 2026 NBER papers and recovers 34% of the papers Pangram 3
flags in 2026 (Pangram 3 itself misses roughly a third of what Pangram 4 finds). It is a
conservative instrument: a flag is strong evidence, a non-flag is weak evidence.

## Install

```bash
pip install git+https://github.com/paulgp/econ-ai-detector
```

The DistilRoBERTa weights (330 MB) download from Hugging Face on first use. To use a
local copy instead, pass `--nn /path/to/dir` or `Detector(nn_path=...)`.

## Use

```bash
econ-ai-detect paper.pdf                 # extract text, drop references, gate non-prose, score
econ-ai-detect intro.txt --no-gate       # already-clean prose: score every window
econ-ai-detect paper.pdf --json          # per-window scores as JSON
econ-ai-detect paper.pdf --target-fpr 0.01
```

```python
from econ_ai_detector import Detector
det = Detector()                         # or Detector(nn_path="/local/dir")
res = det.score_pdf("paper.pdf")
res.flagged, res.lr_k2, res.nn_k2, res.windows
```

## What you should know before trusting a number

- **Calibrated to economics working-paper prose.** The thresholds were set on pre-2020
  NBER and journal text and verified on 2022 NBER papers. On other registers (abstracts,
  referee reports, other fields, non-native prose) the false-positive rate is unknown.
  Recalibrate on known-human text from your own register before reporting a rate.
- **Score body prose.** The census scored an introduction and one numbered body section
  per paper. Front matter, tables, equations and references are out of register; the
  prose gate removes most of it from PDFs, but the cleanest input is the prose itself.
- **Window resolution is 250 words.** Short texts (< 2 windows) cannot trip the k = 2 rule.
- **It is a floor.** Trained on mirrors from one open model (mid-2026), it generalises to
  unseen frontier models only partly and loses ground as their style drifts. Light AI
  editing is largely invisible to it.
- **Numbers are masked.** Digits are replaced by `0` before scoring, so the model cannot
  key on vintage.

## Reproducibility

Scores from this package match the window-score dumps used in the pangram-nber analysis
to 1e-5 (LR) and 1e-3 (NN margin, hardware-dependent). Training scripts, mirrors,
reinforcement rounds and the full evaluation live in the pangram-nber repository
(`scripts/toy_v3_*.py`, `scripts/toy_ultralow.py`).

## Citation

Goldsmith-Pinkham, Paul (2026). *The Missing AI Paper Boom* (working paper) and the
pangram-nber repository. Please cite the paper when using the detector or its numbers.

## License

MIT (code and the logistic-regression weights). The DistilRoBERTa weights on Hugging Face
are released under Apache-2.0, matching the base model.

econ-ai-detector

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

econ-ai-detector

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README