docs-url-discovery-bench

KO | EN

gitlite — search

#mysql #gpt #cpp #proxy #reactjs #pdf #artificial-intelligence #golang #vue #docker

docs-url-discovery-bench

(★ 9)

No description available.

File Explorer

ZIP Download

- breakdown_404.py
- multimodel.py
- stats.py
- - page-inventory.json
  - results.jsonl
- - results.jsonl
- example.json
- full.json
- __init__.py
- audit.py
- exec.py
- grade.py
- probe.py
- proxy.py
- report.py
- run.py
- runners.py
- snapshot.py
- snapshot_crawl.py
- validate.py
.gitignore
LICENSE
README.md
requirements.txt
RESULTS.md

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Benchmarks coding agents on **URL discovery**: given a question, how efficiently does an
agent find the right page on a docs site? It compares four serving formats to measure
whether content negotiation (markdown for agents) and `llms.txt` reduce wasted fetches and
404s.

## Finding

Across 20 Mintlify docs sites and two agents (Claude Code on Sonnet 5, Codex on GPT-5.5),
serving an `llms.txt` index eliminates ~90%+ of 404s versus raw HTML. The main finding is that it's the index, not the markdown, that does it. It replicates on three further models (Opus 4.8, Fable 5, and GPT-5.6), same direction on 19–20 of 20 sites per model. Full numbers and significance are in [`RESULTS.md`](RESULTS.md).

## Arms

| Arm         | Serving behavior                                                |
| ----------- | --------------------------------------------------------------- |
| `html`      | HTML                                                            |
| `md`        | Markdown                                                        |
| `md-link`   | Markdown with the llms.txt pointer                              |
| `md-inline` | Markdown with the full llms.txt content inlined into every page |

All arms serve identical content through a local proxy that rewrites the upstream origin to
itself, so every agent fetch is observable and 404s are counted from proxy logs.

## Metrics

Per arm × agent: **404s per task**, fetches-to-success, tokens, wall time, and, when the
dataset provides expected answers, URL accuracy.

## Setup

```bash
git clone https://github.com/mintlify/url-discovery-bench
cd url-discovery-bench
pip install -r requirements.txt   # requests, numpy, scipy
```

Requires **Python 3.9+**. That's everything Tier 0 needs, but Tiers 1 and 2 additionally need
the [`claude`](https://docs.claude.com/en/docs/claude-code) (Claude Code) and/or
[`codex`](https://github.com/openai/codex) (OpenAI Codex CLI) installed and authenticated.
macOS/Linux only.

## Reproduce

### Tier 0: verify the findings (no API keys, instant)

The raw results of the full run ship in the repo, so every table and p-value is
regenerable without re-running anything.

```bash
python analysis/stats.py data/full/results.jsonl   # aggregate table + site-clustered significance
python -m url_discovery_bench.report data/full      # same aggregate table
python analysis/breakdown_404.py data/full dataset/full.json --per-agent \
  --inventory data/full/page-inventory.json          # 404 composition (proxy logs + frozen inventory, both committed)
python analysis/multimodel.py                       # per-model table across all five models
python analysis/stats.py data/full/results.jsonl data/multimodel/results.jsonl --by-model  # per-model significance
```

### Tier 1: run it on your own docs (live, minutes)

Point the benchmark at any docs site and watch the 404/fetch/token difference across
formats. No ground truth required. `expected` answers are optional; omit them and you get
the efficiency metrics (accuracy shows `n/a`), or add them to also grade correctness.

```bash
cp dataset/example.json dataset/mysite.json
# edit mysite.json: set target, basePath, rewriteOrigins, and a few questions
python -m url_discovery_bench.probe https://docs.mysite.com https://mysite.com/docs   # confirm it's a Mintlify site
python -m url_discovery_bench.run --dataset dataset/mysite.json --agents claude --snapshot-mode record --job mysite
python -m url_discovery_bench.report jobs/mysite
```

`--snapshot-mode record` measures against the live site while freezing every response to
`snapshots/` so the run can later be replayed deterministically (`--snapshot-mode replay`).

### Tier 2: the full study (hours, ~$100, both CLIs)

20 sites × 5 questions × 4 arms × 2 agents × 3 attempts = 2,400 attempts, and re-crawls
~3 GB of snapshots. Exits non-zero while attempts remain, so wrap it in a retry loop:

```bash
until python -m url_discovery_bench.run --dataset dataset/full.json --attempts 3 --job full \
  --agents claude,codex --concurrency 8 --snapshot-mode record; do sleep 120; done
```

## Contamination gate

Every attempt's transcript is audited for commands containing URLs whose host is not the
proxy (`url_discovery_bench/audit.py`). Any hit hard-fails the attempt: it is recorded as an
`error: "contaminated"` row with the offending URLs, excluded from the report, and retried
on resume. Claude is additionally tool-restricted to `Bash(curl:*)` with WebFetch/WebSearch
disallowed and MCP servers disabled; Codex has no per-tool allowlist (its sandbox needs
network to reach the proxy), so the audit gate is the enforcement there. Commands whose URL
can't be statically resolved (e.g. `curl "$u"` in a loop) are counted in the report's
`unaudited` column rather than silently passed; the audit's scope and limits are spelled
out in [`RESULTS.md`](RESULTS.md#limitations).

## Snapshots

`python -m url_discovery_bench.snapshot_crawl --dataset dataset/<file>.json` crawls every page each site
declares (llms.txt + sitemap, both variants) into `snapshots/`. Three proxy modes:

- `live` (default): fetch upstream every request.
- `record`: fetch upstream and persist each response, used for report-quality runs, so
  published numbers reflect real sites while still producing a frozen artifact.
- `replay`: serve purely from disk; no upstream traffic. Deterministic reproduction mode.
  Unknown paths return 404 and are flagged `snapshotMiss` in proxy logs and a
  `snapshotMisses` count per attempt.

Snapshots (~3 GB) and transcripts are gitignored; re-crawl with `python -m url_discovery_bench.snapshot_crawl`.

## Layout

```
url_discovery_bench/proxy.py          # serving proxy: 4 modes, origin rewrite, snapshot record/replay, JSONL log
url_discovery_bench/runners.py        # headless claude (curl-only) + codex exec runners
url_discovery_bench/audit.py          # contamination audit: external-host fetch detection
url_discovery_bench/grade.py          # URL normalization + answer grading (expected optional)
url_discovery_bench/run.py            # orchestrator: task x arm x agent x attempt, one proxy per attempt
url_discovery_bench/report.py         # per-arm aggregate table
url_discovery_bench/snapshot.py       # snapshot read/write + live fetch
url_discovery_bench/snapshot_crawl.py # crawl a site's pages into snapshots/
url_discovery_bench/probe.py          # check whether a site negotiates markdown + serves llms.txt
analysis/stats.py     # aggregate + site-clustered paired significance (Wilcoxon + paired t); --by-model to group by model
analysis/multimodel.py # per-model per-arm table across the core + replication runs
analysis/breakdown_404.py # classify each 404 against the site's real page list (reads <job>/logs)
dataset/              # sites + question -> expected-path ground truth; example.json is a template
data/full/            # committed core-run artifacts (sonnet-5 + gpt-5.5, record mode): results.jsonl, proxy logs, frozen page inventory
data/multimodel/      # committed replication results (results.jsonl): opus-4.8, fable-5, gpt-5.6, replay mode
jobs/                 # run outputs (gitignored): results.jsonl, proxy logs, transcripts
```

## Notes

- Web search and non-curl tools are disabled; the prompt confines the agent to the proxy
  origin and transcripts are audited for external fetches. Answers pointing at any host
  other than the attempt's proxy origin grade incorrect (`offProxy`); a memorized
  production URL never counts as discovery.
- The proxy origin (`127.0.0.1:<port>`) prevents reuse of memorized URLs, but questions
  should still avoid leaking the page slug, and a dataset should span many sites.
- Upstream content can change between runs; run all arms of a comparison in the same
  session, or use `record`/`replay` to freeze content before a report-quality run.
- All fetches go through `curl`, so every request is one observable, countable hit through
  the proxy. That makes the _absolute_ 404 counts method-dependent (a fetch tool that
  converts HTML to markdown client-side would show a smaller absolute gap), but the
  _relative_ index win (`html → md-link`) and its direction are unaffected, and curl-only if
  anything understates HTML's disadvantage rather than inflating it.
- Grading is exact normalized-path match against `expected`. If a question has more than one
  valid answer page, list every acceptable path in `expected` or correct answers score wrong.

## License

MIT, see [`LICENSE`](LICENSE).

docs-url-discovery-bench

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

docs-url-discovery-bench

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README