SkillSmith

KO | EN

gitlite — search

#javascript #ai #llm #golang #android #docker #macos #windows #cli #kubernetes #awesome-list #security #ios #open-source

(★ 30)

SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems

SkillSmith Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Skills and tools that co-evolve from their own failures. 
 A filesystem-based agent that forges reusable skills and tools, tests them, and keeps only what works.

---

## What is SkillSmith?

SkillSmith runs an agent on your tasks, watches where it fails, and **evolves the
agent's skills and tools together** to fix those failures — then validates every
change before keeping it. No git branches, no manual prompt tuning: every candidate
agent is a folder under `.skillsmith/states/`, scored on your data, and kept only if
it actually improves.

<img src="img/main_loop.png" alt="SkillSmith co-evolution loop" width="640">

 
1. **Task arrives** — the agent attempts your questions.
2. **Skill execution** — it applies the skills it has.
3. **Tool execution** — skills call deterministic Python tools.
4. **Reflect & co-evolve** — failures are diagnosed; skills *and* tools are edited or created together. Past failures are remembered as **anti-patterns** so the same mistake isn't proposed twice.
5. **Better agent** — the validated candidate joins the frontier and takes on harder tasks.

## Quick Start

```bash
cd SkillSmith
python -m venv .venv && source .venv/bin/activate
pip install -e .

cp .env.example .env          # then put your key in .env as LLM_API_KEY
skillsmith init               # scaffold .skillsmith/ (task.md, data/, base state)
skillsmith run                # evolve over your data
skillsmith list               # see states and the current frontier
```

Tune any run from the command line — no need to open `config.toml`:

```bash
skillsmith run --iterations 3
skillsmith run --set llm.model=gpt-4o --set evolution.candidates_per_parent=3
skillsmith config             # preview the resolved config and catch problems
```

## Models & Providers

Put one key in `.env` as `LLM_API_KEY` — it works for whichever provider you pick.
Switching models is usually just `provider` + `base_url` + `model`:

```toml
[llm]
provider = "openai"                                   # OpenAI-compatible /chat/completions
base_url = "https://api.openai.com/v1"
model    = "gpt-4o"
```

| Provider | `provider` | `base_url` example |
|----------|------------|--------------------|
| OpenAI / vLLM / OpenRouter / Together | `openai` | `https://api.openai.com/v1` |
| Anthropic (`/messages`) | `anthropic` | `https://api.anthropic.com/v1` |
| Xiaomi — OpenAI format | `openai` | `https://token-plan-sgp.xiaomimimo.com/v1` |
| Xiaomi — Anthropic format | `anthropic` | `https://token-plan-sgp.xiaomimimo.com/anthropic/v1` |

> Prefer a provider-specific key? Set `api_key_env = "OPENAI_API_KEY"`.
> Run `skillsmith config` to see the resolved settings and validate the key, data, and provider.

## Examples

Three ready-to-run projects live under [`examples/`](examples/):

| Example | Task | What it shows |
|---------|------|---------------|
| [`zindle`](examples/zindle) | invented string transform | base **0 → 1.0**: induces a reusable transform skill + tool |
| [`tally`](examples/tally) | letter-sum transform | base **0 → 1.0**, generalizes to unseen words ([RESULTS.md](examples/tally/RESULTS.md)) |
| [`officeqa`](examples/officeqa) | document-grounded numeric QA | a hard, real-world benchmark over Treasury bulletins |

```bash
skillsmith --path examples/tally run    # forge a tool from a handful of examples
```

## How it works

- **Failure-driven** — only failing tasks trigger evolution; failures cluster into reusable *capability* hypotheses.
- **Two-stage proposal** — a reflection pass diagnoses the root cause and writes a repair plan, then an implementation pass builds the skill/tool bundle.
- **Generalization-first** — the proposer, reflector, and implementer all carry anti-overfitting rules (an abstraction ladder, the "would this help 10 tasks or just this one?" test, parameterize-don't-hardcode) adapted from EvoSkill and Anthropic's `skill-creator`, so a bundle encodes a reusable capability instead of a single-task patch.
- **Capability memory** — validated capabilities are remembered and *reused / extended / composed* before anything new is created.
- **Pareto frontier** — candidates are kept on a per-task score frontier, and complementary ones can be merged.
- **Ecology** — per-skill utility and pairwise synergy are estimated over time and (optionally) fed back into skill retrieval.

### Validation gates

Every candidate must pass a configurable set of **gates** before it joins the frontier
(skill authoring → tool unit tests → integration on the failed tasks → capability holdout
→ no validation regression). All default to on; the agent-launching ones are the most
expensive, so they're the first to disable for a faster loop:

```toml
[gates]
integration_check = false     # skip the agent run on the failed minibatch
capability_holdout = false    # skip the agent run on the holdout task
```

<details>
<summary>Full gate list &amp; advanced configuration</summary>

```toml
[gates]
anti_pattern_veto = true       # veto proposals matching known anti-patterns
tool_lifecycle = true          # enforce the compose/split lifecycle policy
skill_authoring = true         # validate SKILL.md frontmatter and structure
tool_unit_tests = true         # run each tool's unittest before integration
bundle_repair = true           # re-implement a candidate from its own preflight failures (see bundle_repair_rounds)
integration_check = true       # agent run on the failed minibatch (expensive)
integration_repair = true      # one-shot repair when integration doesn't improve
capability_holdout = true      # agent run on the capability holdout task (expensive)
validation_regression = true   # reject candidates whose validation score regresses
```

**Multi-round tool repair** — when a candidate fails its own skill/tool tests, it is
re-implemented from the failure (with the real source-document excerpts in context) and
re-tested, up to `bundle_repair_rounds` times before rejection. Default `1` (single-shot);
raise it to give the model more rounds to converge on a tool that passes its own general tests:

```toml
[evolution]
bundle_repair_rounds = 3   # default 1
```

**Skill-first mode** (`allow_tool_creation`, default `true`) — when `false`, the proposer may **not
create a new tool**. It diagnoses the *skill* gap first, authors/repairs the SKILL.md, and only
co-evolves a tool that **already exists** (`edit`/`wrap`/`compose`/`split`/`retire`). A missing
capability is encoded as explicit prose steps in the skill rather than a freshly-minted parser — an
anti-overfit lever for document tasks where minted parsers are brittle and overfit one question:

```toml
[evolution]
allow_tool_creation = false   # default true; false = paper-faithful, skill-primary co-evolution
```

**Ecology-aware retrieval** — blend measured ecology into skill selection
(both default to `0.0`, i.e. pure text relevance):

```toml
[ecology]
retrieval_weight = 0.5   # mix each skill's utility into its relevance score
synergy_weight   = 0.3   # boost skills that synergize with already-picked ones
```

**Investigative proposer** — let the proposer read files / grep the project before proposing:

```toml
[evolution]
proposer_investigation = true
proposer_investigation_max_steps = 4
```

**Embedding-backed anti-pattern retrieval** (OpenAI-compatible endpoints only):

```toml
[anti_patterns]
embedding_retrieval = true
embedding_backend = "llm"
embedding_model = "text-embedding-3-small"
embedding_dimensions = 256
```

**Run tasks through opencode** instead of the direct API:

```toml
[runtime]
executor = "opencode"
opencode_model = "xiaomi/mimo-v2.5-pro"
```

</details>

## Scorers

```toml
[scorer]
type = "multi_tolerance"   # contains | exact | numeric_tolerance | multi_tolerance | json_exact
tolerances = [0.05, 0.01, 0.1, 0.0, 0.025]
```

```bash
skillsmith score "2,602" "2602 million" --scorer multi_tolerance   # quick one-off check
```

## Project layout

```text
.skillsmith/
 config.toml task.md data/{train,val}.csv
 states/
 base/ iter-001/ ... # each state = a full agent
 program.yaml manifest.json
 skills/<name>/SKILL.md
 tools/<name>/{tool.py,test_tool.py}
 frontier.json scores.json ecology.json
 capabilities/ anti_patterns.jsonl
```

## Commands

| Command | What it does |
|---------|--------------|
| `skillsmith init` | scaffold a new project |
| `skillsmith run` | run the evolution loop |
| `skillsmith merge` | merge complementary frontier states |
| `skillsmith list` | list states and the frontier |
| `skillsmith eval <state>` | score a state on validation data |
| `skillsmith diff <a> ` | diff two states |
| `skillsmith config` | show the resolved config and validate it |
| `skillsmith score <expected> <predicted>` | score one answer pair |
| `skillsmith backfill-capabilities` | infer capability memory for older states |

Any `config.toml` value can be overridden with `--set section.key=value` (repeatable).

## Citation

If you use SkillSmith in your research, please cite our paper:

```bibtex
@misc{wei2026skillsmithcoevolvingskillstools,
      title={SkillSmith: Co-Evolving Skills and Tools for Self-Improving Agent Systems}, 
      author={Yangbo Wei and Zhen Huang and Shaoqiang Lu and Junhong Qian and Qifan Wang and Chen Wu and Lei He},
      year={2026},
      eprint={2606.01314},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2606.01314}, 
}
```

SkillSmith

SkillSmith Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

SkillSmith

SkillSmith Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README