eval-harness
My own personal evaluation harness
File Explorer
- ci.yml
- eval_harness_architecture.png
- authorisation.md
- config.md
- helpers.md
- results.md
- evals.example.json
- simple_evals.example.json
- prompt.md
- questions.json
- __init__.py
- eval.py
- integration_tests.rs
- prompt.md
- __init__.py
- eval.py
- encoding_prompt.md
- questions.json
- __init__.py
- eval.py
- broken_crash.py
- broken_wrong.py
- golden.py
- prompt.md
- __init__.py
- eval.py
- __init__.py
- eval.py
- mutants.json
- pristine_module.py
- prompt.md
- __init__.py
- eval.py
- index.html
- broken-nginx.conf
- Dockerfile
- instruction.md
- oracle.sh
- tests.py
- eval.py
- canonical_mapping.csv
- mapping_prompt.md
- __init__.py
- eval.py
- __init__.py
- bug_fix.md
- eval_generator.md
- new_feature.md
- new_feature_authoring_gate.md
- schema_field_mapping.md
- search_with_qa.md
- terminal_task.md
- test_authoring.md
- SKILL.md
- SKILL.md
- SKILL.md
- __init__.py
- settings.py
- .gitkeep
- opencode.json
- Dockerfile
- Dockerfile
- file_helper.py
- naming.py
- tui.py
- evaluation_results.py
- __init__.py
- config.py
- execution.py
- menu.py
- styles.py
- __init__.py
- docker_runner.py
- evals_engine.py
- evaluation_file_protocol.py
- failure_diagnostics.py
- logging_config.py
- models.py
- conftest.py
- test_agent_provisioning_integration.py
- test_docker_images_integration.py
- test_docker_runner_integration.py
- test_evals_engine_integration.py
- test_main_integration.py
- test_repair_nginx_service_integration.py
- conftest.py
- test_docker_runner.py
- test_eval_config.py
- test_evals_engine.py
- test_evaluation_results_repository.py
- test_failure_diagnostics.py
- test_logging_config.py
- test_main.py
- test_naming.py
- test_settings.py
- test_tui_config.py
- test_tui_execution.py
- .gitignore
- .pre-commit-config.yaml
- .python-version
- AGENTS.md
- CLAUDE.md
- LICENSE
- main.py
- pyproject.toml
- README.md
- ruff.toml
- uv.lock
# Use via CDN
jsDelivrjsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.
Command Glossary
Commands referenced in this DOCs, explained below.
act
View Details ▼
act
Execute GitHub Actions locally using Docker.
act {{[-l|--list]}}
List the available jobs:
act
Run the default event:
act {{event_type}}
Run a specific event:
docker build
View Details ▼
docker build
Build an image from a Dockerfile.
docker build .
Build a Docker image using the Dockerfile in the current directory:
docker build {{github.com/creack/docker-firefox}}
Build a Docker image from a Dockerfile at a specified URL:
docker build {{[-t|--tag]}} {{name:tag}} .
Build a Docker image and tag it:
python
View Details ▼
python
Python language interpreter.
python
Start a REPL (interactive shell):
python {{path/to/file.py}}
Execute a specific Python file:
python -i {{path/to/file.py}}
Execute a specific Python file and start a REPL:
uv run
View Details ▼
uv run
Run a command or script in the project environment.
uv run {{path/to/script.py}}
Run a Python script:
uv run {{[-m|--module]}} {{module_name}}
Run a Python module:
uv run {{[-w|--with]}} {{package}} {{command}}
Run a command with additional packages installed temporarily:
uv sync
View Details ▼
uv sync
Update the project's environment to match the lockfile.
uv sync
Sync the project environment with the lockfile:
uv sync --all-extras
Sync and include all optional dependencies:
uv sync --extra {{extra_name}}
Sync with specific optional dependencies:
defaults
View Details ▼
defaults
Read and write macOS user configuration for applications.
defaults read "{{application}}" "{{option}}"
Read system defaults for an application option:
defaults read -app "{{application}}" "{{option}}"
Read default values for an application option:
defaults find "{{keyword}}"
Search for a keyword in domain names, keys, and values: