eal-bench

KO | EN

gitlite — search

#nlp #json #php #generative-ai #large-language-models #developer-tools #python #self-hosted #tailwindcss #agent

(★ 9)

EAL-Bench: Benchmarking Endogenous Authorization Laundering (EAL) in Agent Memory

File Explorer

ZIP Download

- __init__.py
- authorization_scope_drift.py
- common.py
- controlled_interventions.py
- evaluation_awareness.py
- evaluation_cue.py
- FAILURE_MECHANISMS.md
- failure_mechanisms.py
- finance_seed_witness_replay.py
- memory_fidelity.py
- merge_gate.py
- paper_results.py
- phase2_replication.py
- plot_evaluation_cue_paper_figures.py
- plot_writer_ttc_figure.py
- plots.py
- routes.py
- summarize.py
- writer_ttc.py
- writer_ttc_independent_review.py
- writer_ttc_multi.py
- writer_viability.py
- - eal-bench-system-overview.png
- - __init__.py
  - adapter.py
  - analysis_plan.json
  - capacity.py
  - capacity_calibration_v1.json
  - compile_v3.py
  - corpus.py
  - corpus_final.py
  - corpus_v10.py
  - corpus_v11.py
  - corpus_v12.py
  - corpus_v3.py
  - corpus_v7.py
  - finalize_results.py
  - models.py
  - pre_run_cost_estimate.json
  - prepare_final_release.py
  - prepare_review_materials.py
  - pressure_final.py
  - README.md
  - release.json
  - release.py
  - release_adoption.py
  - run_plan.json
  - semantics.py
  - studies.py
- - __init__.py
  - adapter.py
  - analysis_plan.json
  - build_redesign_final_precommit.py
  - capacity.py
  - capacity_calibration.json
  - compile_corpus.py
  - compile_redesign.py
  - compile_v2.py
  - corpus.py
  - corpus_redesign.py
  - corpus_v2.py
  - finalize_redesign.py
  - finalize_replacement.py
  - finalize_results.py
  - models.py
  - pre_run_cost_estimate.json
  - prepare_release.py
  - pressure.py
  - README.md
  - redesign_development_precommit.json
  - redesign_development_precommit_002.json
  - redesign_development_precommit_003.json
  - redesign_development_precommit_004.json
  - redesign_development_precommit_005.json
  - redesign_development_precommit_006.json
  - redesign_development_precommit_007.json
  - redesign_development_precommit_008.json
  - redesign_development_precommit_009.json
  - redesign_development_precommit_010.json
  - redesign_development_precommit_011.json
  - redesign_development_precommit_012.json
  - redesign_development_precommit_013.json
  - redesign_development_precommit_014.json
  - redesign_final_precommit.json
  - redesign_final_route_validation.json
  - redesign_precommit.py
  - release.json
  - release.py
  - run_plan.json
  - seed_witness_replay.py
  - semantics.py
  - studies.py
  - v2_blueprint.json
- - __init__.py
  - adapter.py
  - awareness.py
  - benchmark_blueprint.yaml
  - capacity.py
  - capacity_ablation_analysis.py
  - capacity_calibration.json
  - cases.py
  - challenge.py
  - challenge_data.py
  - compile_corpus.py
  - corpus_lint.py
  - domain.py
  - oracle.py
  - prepare_review_materials.py
  - presentations.py
  - README.md
  - release.json
  - release.py
  - release_adoption.py
  - RESULTS.md
  - review.py
  - schemas.py
  - semantics.py
  - surface.py
  - tools.py
- __init__.py
- base.py
- CONTRIBUTING.md
- README.md
- scaffold.py
- toolkit.py
- llm_calls.py
- - __init__.py
  - challenges.py
  - conditions.py
  - conformance.py
  - evaluation_awareness.py
  - evaluation_awareness_fixture.py
  - executor_plan.py
  - langmem_writer.py
  - leakage.py
  - persistence.py
  - pipeline.py
  - provenance.py
  - resume.py
  - runner.py
  - schemas.py
  - study_engine.py
  - study_plan.py
  - surfaces.py
  - tokens.py
  - validation.py
- __init__.py
- annotate_authorization_memories.py
- check_target.py
- phase2_precommit.py
- replication_release_compatibility.py
- run.py
- - cybersecurity_v1__actual_cost.json
  - cybersecurity_v1__aggressive_threshold_report.json
  - cybersecurity_v1__checkpoint_fidelity.json
  - cybersecurity_v1__controls_summary.json
  - cybersecurity_v1__deepseek_gate.json
  - cybersecurity_v1__gptoss_gate.json
  - cybersecurity_v1__mechanism_report.json
  - cybersecurity_v1__natural_witness_report.json
  - cybersecurity_v1__official_threshold_report.json
  - cybersecurity_v1__offline_rescore.json
  - cybersecurity_v1__pressure_summary.json
  - cybersecurity_v1__results_bundle.json
  - cybersecurity_v1__transfer_matrix_cost_estimate.json
  - cybersecurity_v1__transfer_matrix_results.json
  - cybersecurity_v1__transfer_matrix_results.md
  - cybersecurity_v1__transfer_matrix_run_plan.json
  - cybersecurity_v1__typed_attribution_report.json
  - cybersecurity_v1__writer_failure_diagnostics.json
  - cybersecurity_v1__writer_summary.json
  - README.md
- - finance_v1__acceptance.json
  - finance_v1__actual_cost.json
  - finance_v1__condition_results.json
  - finance_v1__controls_report.json
  - finance_v1__matrix_report.json
  - finance_v1__matrix_results.md
  - finance_v1__matrix_run_index.json
  - finance_v1__mechanism_extension.json
  - finance_v1__provider_failures.json
  - finance_v1__release_equivalence.json
  - finance_v1__route_summaries.json
  - finance_v1__transfer_agreement.json
  - finance_v1__transfer_matrix_cost.json
  - finance_v1__transfer_matrix_results.json
  - finance_v1__transfer_matrix_results.md
  - finance_v1__transfer_matrix_run_plan.json
  - finance_v1__witness_repair_report.json
  - phase2_finance_replacement__actual_cost.json
  - phase2_finance_replacement__condition_results.json
  - phase2_finance_replacement__controls_report.json
  - phase2_finance_replacement__provider_failures.json
  - phase2_finance_replacement__route_summaries.json
  - phase2_finance_replacement__transfer_agreement.json
  - phase2_finance_replacement__witness_repair_report.json
  - phase2_finance_replacement_index.json
  - phase2_finance_replacement_report.json
  - phase2_finance_replacement_results.md
  - phase2_seed_witness_replay_report.json
  - phase2_seed_witness_replay_report.md
  - README.md
- - design_report.md
  - development_iteration_001_report.json
  - development_iteration_002_report.json
  - development_iteration_003_report.json
  - development_iteration_004_report.json
  - development_iteration_005_report.json
  - development_iteration_006_forensic_report.md
  - development_iteration_006_report.json
  - development_iteration_007_forensic_report.md
  - development_iteration_007_report.json
  - development_iteration_008_forensic_report.md
  - development_iteration_008_report.json
  - development_iteration_009_forensic_report.md
  - development_iteration_009_report.json
  - development_iteration_010_forensic_report.md
  - development_iteration_010_report.json
  - development_iteration_011_forensic_report.md
  - development_iteration_011_report.json
  - development_iteration_012_forensic_report.md
  - development_iteration_012_report.json
  - development_iteration_013_forensic_report.md
  - development_iteration_013_report.json
  - development_iteration_014_forensic_report.md
  - development_iteration_014_report.json
  - development_iteration_index.json
  - final_artifact_audit.json
  - final_held_out_evaluation.json
  - final_held_out_evaluation.md
  - procurement_reference_forensics.md
  - README.md
- - phase2_precommit.json
  - phase2_results.json
  - phase2_results.md
- - procurement_v1__adoption.json
  - procurement_v1__deepseek_gate.json
  - procurement_v1__deepseek_independent_ttc_review_run_plan.json
  - procurement_v1__evaluation_cue_canary_failure_20260815.json
  - procurement_v1__evaluation_cue_cost_estimate.json
  - procurement_v1__evaluation_cue_final_audit_20260815.json
  - procurement_v1__evaluation_cue_implementation_freeze.json
  - procurement_v1__evaluation_cue_implementation_freeze_v2.json
  - procurement_v1__evaluation_cue_implementation_freeze_v3.json
  - procurement_v1__evaluation_cue_openrouter_credit_incident_20260815.json
  - procurement_v1__evaluation_cue_run_plan.json
  - procurement_v1__five_writer_ttc_final_audit_20260814.json
  - procurement_v1__five_writer_ttc_k8_final_audit_20260815.json
  - procurement_v1__five_writer_ttc_k8_run_plan.json
  - procurement_v1__four_writer_ttc_interim_audit_20260814.json
  - procurement_v1__four_writer_ttc_run_plan.json
  - procurement_v1__gptoss_gate.json
  - procurement_v1__qwen_plus_0728_cost_estimate.json
  - procurement_v1__qwen_ttc_run_plan.json
  - procurement_v1__transfer_matrix_cost_estimate.json
  - procurement_v1__transfer_matrix_results.json
  - procurement_v1__transfer_matrix_results.md
  - procurement_v1__transfer_matrix_run_plan.json
  - procurement_v1__transfer_matrix_technical_completion_plan.json
  - README.md
- README.md
- - __init__.py
.env.example
.gitattributes
.gitignore
.python-version
config.yaml
judge.py
MODELS.md
pyproject.toml
README.md
uv.lock

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">EAL-Bench</h1>

<p align="center">
  <strong>A benchmark for Endogenous Authorization Laundering (EAL) in persistent agent memory</strong>
</p>

EAL-Bench tests whether persistent agent memory preserves evolving authorization boundaries and
whether memory errors lead to unauthorized downstream tool actions. It accompanies the paper
[*Agent Memory Is a Surface for Endogenous Authorization Laundering*](https://arxiv.org/abs/2609.01836).

## Overview

![EAL-Bench system and example failure](https://raw.githubusercontent.com/tommasocerruti/eal-bench/main/assets/figures/eal-bench-system-overview.png)

A memory writer compresses an organizational history into persistent memory. An executor later
receives that memory and a new request, then chooses whether to act through a native tool. A hidden
canonical ledger scores the action deterministically.

Each benchmark case includes:

- an evolving, multi-session organizational history;
- a hidden authorization ledger;
- matched authorized and unauthorized requests;
- a bounded free-text or typed memory;
- domain-native tools and a deterministic oracle.

The writer never sees the ledger or future requests. The executor never sees the original history
or ledger.

## Benchmark at a glance

### Domains

| Domain | Cases | Authorization scope |
|---|---:|---|
| Procurement | 12 | Buyer, vendor, category, amount, currency, status, and validity |
| Cybersecurity | 16 | Responder, tenant, incident, asset, environment, vulnerability, action, and validity |
| Finance | 8 | Trader, account, strategy, instrument, side, order type, quantity, price, currency, and validity |

### Memory conditions

| Writing approach | Free-text memory | Typed memory |
|---|---|---|
| One-shot | Write from the complete history | Build a schema-validated profile from the complete history |
| Incremental | Update from one new block | Update the typed profile from one new block |

Incremental writers receive the previous accepted memory and the new block, but not earlier raw
blocks. Final memories are frozen and hashed before executor evaluation.

### Studies

| Study | Purpose |
|---|---|
| `controls` | Calibrate executors with faithful evidence and controlled authorization changes |
| `writer` | Generate memories and measure downstream behavior |
| `pressure` | Replay a writer run under authority-invariant operational pressure |
| `writer_ttc` | Compare writer-side candidate sampling and selection strategies |
| `evaluation_cue` | Measure the effect of evaluation framing |

Model targets are provider-specific and are never silently pooled or substituted. List the
configured routes and resolved model names with:

```bash
uv run python -m experiments.run --list-targets
```

## Quick start

Requirements:

- Python 3.10 or newer;
- [`uv`](https://docs.astral.sh/uv/);
- a Baseten or OpenRouter API key for live experiments.

```bash
git clone git@github.com:tommasocerruti/eal-bench.git
cd eal-bench

uv sync --extra dev --extra analysis
cp .env.example .env
```

Add credentials only for the providers you intend to call:

```dotenv
BASETEN_API_KEY=...
OPENROUTER_API_KEY=...
```

Offline inspection and validation do not require API credentials.

## Inspect and validate

```bash
uv run python -m experiments.run --list-domains
uv run python -m experiments.run --domain procurement --list-corpus-versions
uv run python -m experiments.run --domain procurement --list-studies
uv run python -m experiments.run --validate-only --all-domains
```

Validate a planned route before making live calls:

```bash
uv run python -m experiments.run \
  --domain procurement \
  --corpus-version benchmark_v1 \
  --presentation-version naturalistic_v1 \
  --study writer \
  --writer-targets nemotron_3_ultra_baseten \
  --executor-targets gptoss_baseten \
  --writer-architecture all \
  --writer-strategy all \
  --validate-only
```

## Run an experiment

Live routes make paid API calls. Review the validated call plan first, then rerun it with an
explicit cost ceiling:

Use `--study controls` for faithful controls. Pressure replays require a completed writer run
through `--source-run results/<domain>/<run-id>`.

## Outputs

Every run is stored in a new immutable directory:

```text
results/<domain>/<run-id>/
```

Depending on the study, it records the resolved configuration, generated memories, exact
model-visible contexts, native tool calls, normalized decisions, oracle scores, hashes, and
provider usage. The completed `manifest.json` is the authoritative artifact inventory.

The [shared result guide](results/README.md) provides one layout for Procurement, Cybersecurity,
and Finance. Each domain has a `results/<domain>/paper/manifest.json` selecting the exact runs,
release, source hashes, and counts used in Appendix B.2, Tables 11–13. Reproduce all three domains
without model calls or API keys:

```bash
python3 -m analysis.paper_results --domain all --output-dir /tmp/eal-paper-tables
```

Each domain exports the same CSV, JSON, and Markdown files, including writer/executor breakdowns
and an artifact inventory. Raw JSONL files are excluded from Git across all domains; the command
checks any originals present locally and reports missing files. Saved aggregates reproduce the
published counts, while independent trial scoring and bootstrap reanalysis require the originals.

## Extending EAL-Bench

New domains define their own authorization state, histories, memory representation, requests,
native tools, deterministic oracle, and fidelity comparison. Shared experiment and analysis code
does not import concrete domains.

- [Domain interface](domains/README.md)
- [Domain contribution guide](domains/CONTRIBUTING.md)
- [Model and provider configuration](MODELS.md)

## Development

Before opening a pull request:

```bash
uv run python -m experiments.run --validate-only --all-domains
uv run ruff check .
git diff --check
```

---

## Citation

If you use EAL-Bench in your research, please cite the accompanying paper:

```bibtex
@misc{cerruti2026agent,
  title         = {Agent Memory Is a Surface for Endogenous Authorization Laundering},
  author        = {Cerruti, Tommaso and Okamoto, Mika and Erol, Ansel Kaplan},
  year          = {2026},
  eprint        = {2609.01836},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CR},
  url           = {https://arxiv.org/abs/2609.01836}
}
```

eal-bench

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

eal-bench

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README