mcp-load-lab - GitLite

KO | EN

gitlite — search

#reactjs #animation #devops #pdf #golang #android #react #rag #shell #python

mcp-load-lab

(★ 21)

Load and latency harness for Model Context Protocol (MCP) servers. Drives real tools/call traffic at configurable concurrency, measures latency percentiles and throughput, and fails CI when a server misses its performance budget. Stdlib-first, zero required dependencies for stdio.

파일 탐색기

ZIP 다운로드

- echo_server.py
- scenario.json
- __init__.py
- __main__.py
- cli.py
- client.py
- config.py
- gates.py
- metrics.py
- report.py
- runner.py
- test_smoke.py
.gitignore
LICENSE
pyproject.toml
README.md

# CDN으로 사용하기

jsDelivr

jsDelivr는 공개 GitHub 리포지토리를 별도 설정 없이 CDN으로 즉시 서빙합니다. 버전과 파일을 고르면 웹페이지에 바로 붙일 수 있는 링크와 예시 코드가 만들어집니다.

버전

파일

링크

예시

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

A load and latency harness for **Model Context Protocol (MCP)** servers.

Most MCP tooling checks whether a server is *correct*: does it follow the spec,
is its tool surface stable, is it safe to deploy. `mcp-load-lab` answers a
different question: **how does the server behave under concurrent load?** It
drives real `tools/call` traffic at a chosen concurrency, measures latency
percentiles and throughput, and fails your CI build when a server misses its
performance budget.

It has **no required dependencies** for the stdio transport (Python standard
library only). HTTP transport adds a single optional dependency.

## Install

```bash
pip install mcp-load-lab            # stdio transport, zero deps
pip install "mcp-load-lab[http]"    # adds streamable-HTTP support (httpx)
```

Or from a clone:

```bash
pip install -e .
```

## Quick start

Every example below works against the bundled demo server in `examples/`,
so you can try the tool before pointing it at your own.

List a server's tools:

```bash
mcp-load-lab tools --stdio "python examples/echo_server.py"
```

Ad-hoc load run, no config file needed:

```bash
mcp-load-lab quick \
  --stdio "python examples/echo_server.py" \
  --tool sleep --arg ms=20 \
  --concurrency 32 --duration 5 --warmup 1
```

```
phase    conc  reqs  rps     ok%    mean  p50   p90   p95   p99   max
-------  ----  ----  ------  -----  ----  ----  ----  ----  ----  ----
warmup*  32    1485  1454.5  100.0  21.7  21.6  22.4  22.5  23.5  23.6
load     32    4512  1498.3  100.0  21.3  21.3  21.9  22.1  22.4  22.9

(latencies in ms; * = warmup, excluded from gates)
```

## Scenario files

For repeatable runs and CI, describe the whole test in a JSON or TOML file:

```jsonc
{
  "name": "echo-sleep-ramp",
  "transport": {
    "type": "stdio",
    "command": "python",
    "args": ["examples/echo_server.py"]
  },
  "calls": [
    { "tool": "sleep", "arguments": { "ms": 20 }, "weight": 3 },
    { "tool": "echo",  "arguments": { "text": "ping" }, "weight": 1 }
  ],
  "phases": [
    { "name": "warmup", "concurrency": 4,  "duration_s": 1, "warmup": true },
    { "name": "c8",     "concurrency": 8,  "duration_s": 3 },
    { "name": "c32",    "concurrency": 32, "duration_s": 3 },
    { "name": "c64",    "concurrency": 64, "duration_s": 3 }
  ],
  "timeout_s": 10,
  "gates": {
    "max_p95_ms": 120,
    "max_error_rate": 0.01,
    "min_throughput_rps": 200
  }
}
```

Run it:

```bash
mcp-load-lab run examples/scenario.json --json report.json
```

- **`calls`** are chosen per request by `weight` (weighted random). Use this to
  model a realistic mix of tools.
- **`phases`** run in order. A `warmup` phase is measured but excluded from the
  gates. Describe a concurrency ramp by listing several phases, each with a
  higher `concurrency`.
- **`gates`** are optional thresholds. If any gate fails, the aggregate of all
  non-warmup phases is reported as `FAILED` and the process exits non-zero.

## Transports

- **stdio** launches your server as a subprocess and speaks newline-delimited
  JSON-RPC over its pipes. Concurrent requests are multiplexed over the single
  connection with distinct ids, so a server that handles requests concurrently
  shows real concurrency.

```jsonc
  { "type": "stdio", "command": "python", "args": ["server.py"], "env": {}, "cwd": "." }
  ```

- **http** POSTs JSON-RPC to a streamable-HTTP endpoint using a pooled client
  (needs the `http` extra). Session ids returned on `initialize` are reused.

```jsonc
  { "type": "http", "url": "http://localhost:8000/mcp", "headers": {} }
  ```

## CI gating

`mcp-load-lab` exits `0` when all gates pass and `1` when any gate fails, so it
drops straight into a pipeline as a performance regression check:

```yaml
- name: MCP performance budget
  run: mcp-load-lab run perf/scenario.json --json perf/report.json
```

Gates can also be set or overridden on the command line:

```bash
mcp-load-lab run perf/scenario.json \
  --max-p95 150 --max-error-rate 0.01 --min-rps 300
```

## Metrics

Per phase and for the non-warmup aggregate the harness reports request count,
throughput (requests/second over wall-clock time), success rate, and latency
`min / mean / p50 / p90 / p95 / p99 / max` in milliseconds. Percentiles use
linear interpolation over the sorted per-request latencies. Failures are
broken out by kind: `tool_error` (server returned `isError`), `rpc_error`
(JSON-RPC error), `timeout`, and `exception`.

## Demo server tools

`examples/echo_server.py` is a dependency-free async MCP server exposing four
tools chosen to exercise different load shapes: `echo` (trivial), `sleep(ms)`
(I/O-bound), `fib(n)` (CPU-bound), and `flaky(rate)` (fails with a given
probability, for testing the error path and error-rate gates).

## Load model

The engine is **closed-loop**: each phase runs a fixed number of concurrent
workers, and each worker issues one call, waits for the response, and
immediately issues the next until the phase's `duration_s` or `requests` budget
is spent. This measures how a server responds at a specific concurrency level.
Open-loop (fixed target RPS) pacing is on the roadmap.

## License

Apache-2.0. See [LICENSE](LICENSE).