agent-anvil
Agent Anvil is a CI-first eval harness for tool-using AI agents. It runs scenario suites, captures traces, grades agent behavior, clusters failures, and suggests concrete prompt/tool/guardrail fixes.
File Explorer
- bug_report.yml
- feature_request.yml
- scenario_request.yml
- agent-anvil.yml
- ci.yml
- openai-demo.yml
- __init__.py
- canonical.py
- contracts.py
- errors.py
- evidence.py
- identity.py
- io.py
- yaml.py
- __init__.py
- adapter_templates.py
- agent.py
- attestations.py
- benchmark.py
- cli.py
- clustering.py
- config.py
- conformance.py
- contracts.py
- doctor.py
- external.py
- fix.py
- flakiness.py
- fuzzing.py
- grading.py
- ingest.py
- init.py
- jsonpath.py
- leaderboard.py
- learning.py
- mcp_audit.py
- mcp_repair.py
- outcomes.py
- packs.py
- pr_comment.py
- redaction.py
- repair.py
- report.py
- runner.py
- scenario.py
- storage.py
- summary.py
- terminal.py
- trace.py
- trace_bridge.py
- fake_mcp_server.py
- leaderboard-auto-pr-workflow.yml
- leaderboard-index-workflow.yml
- leaderboard-submission-workflow.yml
- mcp-harden-workflow.yml
- pr-comment-workflow.yml
- mcp-tools.json
- evaluator_ablation.csv
- evaluator_ablation.tex
- missed_failures.csv
- outcome_counts.csv
- suite_results.csv
- suite_results.tex
- artifact.md
- limitations.md
- results.json
- results.md
- tables.md
- 2026-05-01-agent-anvil-mvp.md
- 2026-05-22-paper-benchmark-pack.md
- 2026-06-02-product-trust-hardening-v1.md
- 2026-06-04-external-agent-adapters-v1.md
- 2026-06-04-external-agent-conformance-v1.md
- 2026-06-04-fastapi-http-agent-example-v1.md
- 2026-06-04-http-agent-conformance-v1.md
- 2026-06-04-http-agent-endpoint-v1.md
- 2026-06-04-http-init-onboarding-v1.md
- 2026-06-04-node-http-agent-example-v1.md
- 2026-06-04-stable-contracts-v1.md
- 2026-06-05-openai-agents-sdk-adapter-example-v1.md
- 2026-06-05-openai-agents-template-http-pattern-v1.md
- 2026-06-06-http-python-adapter-template-v1.md
- 2026-06-06-langgraph-template-http-pattern-v1.md
- 2026-08-03-leaderboard-artifact-attestation-v1.md
- 2026-08-14-assurance-foundation.md
- 2026-05-22-paper-benchmark-pack-design.md
- 2026-08-03-leaderboard-artifact-attestation-design.md
- 2026-08-14-assurance-foundation-design.md
- adapters.md
- artifacts.md
- assurance-trust.md
- cli.md
- conformance-report.md
- conformance.md
- contracts.md
- demo-repair-plan.md
- demo-report.md
- demo-trace.json
- demo.svg
- engineering.md
- http-fastapi-agent.md
- ingest.md
- init.md
- judges-guide.md
- leaderboard.md
- learn.md
- learned-regression.yaml
- limits.md
- marketplace.md
- mcp-audit.md
- mcp-repair.md
- mcp-tool-safety.yaml
- mcp.md
- node-http-agent.md
- openai-agents-sdk-agent.md
- openai-demo-repair-plan.md
- openai-demo-report.md
- openai-demo-tool-trace.json
- openai-demo-trace.json
- openai-graded-regression-repair-plan.md
- openai-graded-regression-report.md
- openai-graded-regression-results.json
- openai-graded-regression-trace.json
- openai-graded-regression-valid-trace.json
- packs.md
- patched-demo-report.md
- patched-demo-trace.json
- privacy.md
- protocol.md
- refund-agent-fuzzed.yaml
- release-provenance.md
- scenarios.md
- schema-versioning.md
- stability.md
- submission.md
- tool-safety-repair-plan.md
- tool-safety-report.md
- tool-safety-trace.json
- trust.md
- __init__.py
- agent.py
- app.py
- agent.mjs
- package.json
- server.mjs
- __init__.py
- agent.py
- app.py
- __init__.py
- agent.py
- deterministic_agent.py
- openai_agent.py
- patched_agent.py
- system_prompt.md
- tools.py
- __init__.py
- account_admin_agent.py
- data_pipeline_agent.py
- external_jsonl_agent.py
- tool_safety_agent.py
- account_admin_trace.yaml
- data_pipeline_trace.yaml
- external_protocol_trace.yaml
- refund_trace.yaml
- tool_safety_trace.yaml
- paper.yaml
- malformed_agent.py
- missing_final_output_agent.py
- pass_agent.py
- assurance-evidence-record-valid.json
- assurance-release-contract-valid.yaml
- compare-result-valid.json
- doctor-report-valid.json
- leaderboard-artifact-attestation-verification-valid.json
- leaderboard-audit-valid.json
- leaderboard-evidence-index-valid.json
- leaderboard-github-run-verification-valid.json
- leaderboard-index-valid.json
- leaderboard-maintainer-rerun-valid.json
- leaderboard-submission-valid.json
- results-valid.json
- run-manifest-valid.json
- scenario-valid.yaml
- trace-protocol-error.json
- trace-valid.json
- app.py
- leaderboard_view.py
- README.md
- requirements.txt
- main.tex
- README.md
- references.bib
- .gitkeep
- external_jsonl_agent.yaml
- http_fastapi_agent.yaml
- node_http_agent.yaml
- openai_agents_sdk_agent.yaml
- refund_agent.yaml
- refund_agent_patched.yaml
- tool_safety.yaml
- agent-anvil.leaderboard.artifact_attestation_verification.v1.schema.json
- agent-anvil.leaderboard.audit.v1.schema.json
- agent-anvil.leaderboard.evidence_index.v1.schema.json
- agent-anvil.leaderboard.github_run_verification.v1.schema.json
- agent-anvil.leaderboard.index.v1.schema.json
- agent-anvil.leaderboard.maintainer_rerun.v1.schema.json
- agent-anvil.leaderboard.v1.schema.json
- anvil.compare.result.v1.schema.json
- anvil.doctor.report.v1.schema.json
- anvil.results.v1.schema.json
- anvil.run_manifest.v1.schema.json
- anvil.scenario.v1.schema.json
- anvil.trace.v1.schema.json
- assurance.anvil.dev.evidence-record.v1alpha1.schema.json
- assurance.anvil.dev.release-contract.v1alpha1.schema.json
- demo.sh
- conftest.py
- test_canonical.py
- test_errors.py
- test_evidence.py
- test_evidence_graph.py
- test_identity.py
- test_pack_registry.py
- test_public_schemas.py
- test_release_contracts.py
- conftest.py
- test_action_metadata.py
- test_adapter_templates.py
- test_assertions.py
- test_attestations.py
- test_benchmark.py
- test_cli.py
- test_community_docs.py
- test_config.py
- test_conformance.py
- test_contracts.py
- test_demo_script.py
- test_docker_metadata.py
- test_doctor.py
- test_external_agent_protocol.py
- test_fix.py
- test_fuzzing.py
- test_grading.py
- test_http_fastapi_example.py
- test_huggingface_leaderboard_space.py
- test_ingest.py
- test_jsonpath.py
- test_leaderboard.py
- test_learning.py
- test_mcp_audit.py
- test_mcp_repair.py
- test_node_http_example.py
- test_openai_agents_sdk_example.py
- test_openai_grader.py
- test_outcomes.py
- test_packs.py
- test_paper_agents.py
- test_pr_comment.py
- test_repair.py
- test_report_generation.py
- test_runner_and_storage.py
- test_scenario_loader.py
- test_storage.py
- test_support_agent_modes.py
- test_trace_bridge.py
- test_trace_schema.py
- .env.example
- .gitignore
- .python-version
- action.yml
- CONTRIBUTING.md
- docker-compose.yml
- Dockerfile
- LICENSE
- pyproject.toml
- README.md
- SECURITY.md
- uv.lock
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
npm
View Details ▼
npm
JavaScript and Node.js package manager.
Manage Node.js projects and their module dependencies.
Some subcommands such as `install`, `run`, etc. have their own usage documentation.
npm init {{[-y|--yes]}}
Create a `package.json` file with default values (omit `--yes` to do it interactively):
npm {{[i|install]}}
Download all the packages listed as dependencies in `package.json`:
npm {{[i|install]}} {{package_name}}@{{version}}
Download a specific version of a package and add it to the list of dependencies in `package.json`:
uv run
View Details ▼
uv run
Run a command or script in the project environment.
uv run {{path/to/script.py}}
Run a Python script:
uv run {{[-m|--module]}} {{module_name}}
Run a Python module:
uv run {{[-w|--with]}} {{package}} {{command}}
Run a command with additional packages installed temporarily:
uv sync
View Details ▼
uv sync
Update the project's environment to match the lockfile.
uv sync
Sync the project environment with the lockfile:
uv sync --all-extras
Sync and include all optional dependencies:
uv sync --extra {{extra_name}}
Sync with specific optional dependencies:
uvicorn
View Details ▼
uvicorn
Python ASGI HTTP Server, for asynchronous projects.
uvicorn {{import.path:app_object}}
Run Python web app:
uvicorn --host {{localhost}} --port {{8080}} {{import.path:app_object}}
Listen on port 8080 on localhost:
uvicorn --reload {{import.path:app_object}}
Turn on live reload:
