oss-migration-eval - GitLite

KO | EN

gitlite — search

#shell #javascript #terminal #golang #self-hosted #ai-agents #cpp #json #claude-code #artificial-intelligence

oss-migration-eval

(★ 9)

Decide whether to switch an LLM pipeline to a cheaper/open-source model — honest cost-vs-accuracy verdict with error bars. A Claude Code skill.

파일 탐색기

ZIP 다운로드

- candidate-selection.md
- deployment.md
- methodology.md
- metrics.md
- optimization.md
- rigor.md
- triage.md
- build_golden.py
- grade.py
- judge.py
- pick_candidates.py
- report_template.md
- run_sweep.py
.gitignore
LICENSE
README.md
SKILL.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Decide — rigorously — whether to move an expensive LLM pipeline to a cheaper or open-source model.** A Claude Code skill that estimates *cost savings vs accuracy loss*, finds the most probable replacement models, and produces an honest go/no-go verdict with error bars.

The uncomfortable truth this skill is built around: **most eval "winners" are single-sample noise.** A real experiment crowned a decisive 9.00/10 winner off one generation per cell; a 5-seed rigor pass dropped it to 7.00 ± 2.0 — mid-pack in a six-way statistical tie. Within-model variance across seeds nearly equalled the entire between-model ranking. So the honest output is often *"no clear winner — here's the tradeoff, and here are the models you can safely rule out."* Saying that is a success, not a failure.

## What it does

Given a recurring, expensive LLM task (lead scoring, extraction, classification, outbound copy, …), it:

1. **Finds the fattest tasks** worth migrating (pairs with [`token-audit`](https://github.com/BayramAnnakov/token-audit-skill)).
2. **Frames the accuracy metric** — deterministic for structured output, a blind panel for subjective.
3. **Builds a human-validated golden set** on your frontier model, *including hard and false-trigger trap cases*.
4. **Shortlists candidates** live from quality (Artificial Analysis) and usage (OpenRouter) leaderboards — never a hardcoded list.
5. **Sweeps** each candidate provider-pinned across multiple seeds, recording measured cost + latency.
6. **Grades with error bars** and reports **statistically tied vs clearly worse** — you can only reliably name the losers, not a winner.
7. **Optionally optimizes** the gap with a prompt-tuning loop (train/test/val discipline; pairs with `autoresearch`).
8. **Deploys via orchestration** — the smart model orchestrates, the cheap model does the narrow task; local via Ollama; fine-tuning as a last resort.

## Why it's different — the honesty guardrails

Baked in, because a naive eval lies to you:

- **≥ 3 seeds, default 5** — single-seed rankings are noise; every number ships with a 95% CI.
- **"Tied" means *not separable at this n*, not *equivalent*** — it prints the minimum difference detectable at your sample size.
- **Case-clustered statistics** — at temperature 0 the per-seed repeats are near-replicates, so effective *n* = cases, not cells.
- **Provider-pinned to highest precision** — unpinned OpenRouter routing silently serves fp4/int4 vs fp8/bf16 call-to-call; a benchmark lottery.
- **Deterministic checks beat the LLM judge** — a "is this number actually in the input?" check disagreed with the judges' fabrication flags, and was right.
- **Mandatory hard + trap cases** — easy-only golden sets over-claim "solved."
- **Human-validated oracle, enforced** — the scripts refuse to grade against a reference no human signed off on.
- **Refuses bad-fit candidates** — one-off tasks, no checkable output, safety-critical quality bars, or a frontier baseline that isn't itself stable.

## Install

Clone into your Claude Code skills directory:

```bash
git clone https://github.com/BayramAnnakov/oss-migration-eval.git ~/.claude/skills/oss-migration-eval
```

Then just ask Claude Code things like *"can a cheaper model replace my lead-scoring pipeline?"*, *"am I overpaying for this LLM task?"*, or *"evaluate OSS alternatives for X."*

## Quick start (manual harness run)

The `templates/` are scaffolds you copy into a working directory and adapt:

```bash
mkdir eval && cd eval && cp ~/.claude/skills/oss-migration-eval/templates/* .
# 1. write tasks.py (schema documented at the top of build_golden.py)
export OR_KEY=<your-openrouter-key>
python3 build_golden.py        # propose references on the frontier model -> STOP, a human validates them
python3 pick_candidates.py discover   # live open-weight models + pricing
python3 pick_candidates.py pin        # highest-precision provider pins
python3 run_sweep.py           # N-seed, provider-pinned, resumable sweep
python3 grade.py               # deterministic metrics + cost/accuracy tradeoff table (no API calls)
python3 judge.py               # blind multi-family panel for subjective tasks
# then fill templates/report_template.md
```

A full reference run (9 models × 8 cases × 5 seeds + a 3-judge panel) costs about **$1** on OpenRouter. Never cut seeds to save cents.

## Requirements

- [Claude Code](https://claude.com/claude-code)
- Python 3.9+ (standard library only — no dependencies)
- An [OpenRouter](https://openrouter.ai) API key with a few dollars of credit

## Credits

Methodology by **Bayram Annakov**, taught in the AI-Native Product / Productivity course. The rigor guardrails come from a self-red-teamed evaluation whose headline reversed under scrutiny — the story that motivates every rule in [`references/rigor.md`](references/rigor.md).

## License

MIT — see [LICENSE](LICENSE).