vllm-bayesian-autotuner

KO | EN

gitlite — search

#python #rust #css #docker #awesome #tailwindcss #data-science #llm #nodejs #editor

vllm-bayesian-autotuner

(★ 9)

No description available.

File Explorer

ZIP Download

- - __init__.py
  - benchmark.py
  - catalog.py
  - connection.py
  - dto.py
  - jobs.py
  - memory.py
  - runs.py
  - session.py
  - tuning.py
  - views.py
- __init__.py
- bayes_opt.py
- benchmark.py
- client.py
- config_space.py
- detector.py
- energy.py
- metrics.py
- model_info.py
- orchestration.py
- prompts.py
- remote_process.py
- reporting.py
- run_logging.py
- ssh_client.py
- system_info.py
- hypernova-60b.json
- qwen36-35b-a3b-1gpu.json
- qwen36-35b-a3b-2gpu-dp.json
- qwen36-35b-a3b-2gpu.json
- qwen36-35b-a3b-rich-spec.json
- qwen36-smoke-spec.json
- - code.jsonl
  - complex_reasoning.jsonl
  - data_processing.jsonl
  - long_context.jsonl
  - mcp_tool_use.jsonl
  - rag_context.jsonl
  - simple_qa.jsonl
- 01-introduccion-y-motivacion.md
- 02-background-vllm-pagedattention.md
- 03-background-bayesian-optimization.md
- 04-arquitectura-del-sistema.md
- 05-espacio-de-configuracion.md
- 06-diseno-bateria-prompts.md
- 07-metricas-y-funcion-objetivo.md
- 08-bayesian-optimization-vllm.md
- 09-plan-evaluacion-experimental.md
- 10-limitaciones-trabajo-futuro.md
- 11-diseno-experimental-y-preparacion.md
- 12-logging-y-trazabilidad-experimental.md
- 13-conexion-ssh-y-gestion-remota.md
- 14-ampliacion-espacio-config-paralelismo-speculative.md
- 15-early-stopping-convergencia.md
- 16-optimizacion-multiobjetivo-pareto.md
- 17-caso-qwen36-35b-a3b-diseno.md
- 18-trabajo-relacionado-y-diferenciacion.md
- referencias.md
- 1_VRAM_Calculator.py
- 2_Dashboard.py
- .gitkeep
- __init__.py
- analyze_experiment.py
- anonymize_results.py
- build_prompt_banks.py
- build_quality_report.py
- build_speed_comparison_report.py
- build_speed_report.py
- merge_and_score_comparison.py
- quality_eval.py
- run_autotune_optuna.py
- run_benchmark_once.py
- run_benchmark_sweep.py
- __init__.py
- test_bayes_opt.py
- test_cli_config.py
- test_client.py
- test_config_space.py
- test_detector.py
- test_hybrid_memory.py
- test_integration_fakeserver.py
- test_mamba_knobs.py
- test_metrics.py
- test_metrics_extended.py
- test_model_info.py
- test_orchestration.py
- test_prompts.py
- test_remote_process.py
- test_reporting.py
- test_run_logging.py
- test_scheduler_knobs.py
- test_service.py
- test_service_boundary.py
- test_service_memory_runs.py
- test_ssh_client.py
- test_system_info.py
.gitignore
app.py
LICENSE
pyproject.toml
README.md
REPRODUCE.md
test_model.py

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Automatic configuration tuning for vLLM inference servers via Bayesian Optimization.**

This project autotunes [vLLM](https://github.com/vllm-project/vllm) serving configurations by exploring a 24-dimensional parameter space with Tree-structured Parzen Estimator (TPE) optimization. It converges to high-performing settings in 50–100 trials, avoiding the costly trial-and-error of manual tuning.

Each trial restarts `vllm serve` with a new configuration, runs an asynchronous benchmark over representative prompts, measures throughput, latency, and goodput, and feeds the resulting score back into the optimizer. Infeasible configurations are pruned analytically before any GPU time is spent.

## At a glance

| Feature | Status |
|---|---|
| TPE-based Bayesian Optimization (Optuna) | ✅ |
| Random search baseline | ✅ |
| Multi-objective optimization (Pareto fronts) | ✅ |
| Infeasibility pruning (analytical memory model) | ✅ |
| SSH-driven server lifecycle management | ✅ |
| Hybrid attention memory model (Mamba + KV) | ✅ |
| Speculative decoding integration | ✅ |
| Streaming metrics (TTFT, TPOT, ITL, goodput) | ✅ |
| Energy monitoring (tokens/joule) | ✅ |
| Web UI (Streamlit) | ✅ |
| Exhaustive run logging | ✅ |

## Verified results

Experiments on **NVIDIA H100 NVL** (93.58 GiB) with **Qwen3.6-35B-A3B** (35B MoE, hybrid attention, FP8):

| Experiment | Strategy | Seeds | Score (mean ± σ) | Throughput (tok/s) |
|---|---|---:|---:|---:|
| Rich space (20+ dims, speculative decoding) | **TPE** | 2 | **2,594 ± 3** | **3,704 ± 3** |
| | Random | 4 | 2,462 ± 54 | 3,591 ± 67 |
| Flat space (6 dims) | **TPE** | 1 | **4,630** | **3,677** |
| | Random | 1 | 4,612 | 3,664 |

**Key findings:**
- TPE outperforms Random by **+5.4%** in objective score and **+3.1%** in peak throughput on complex search spaces.
- TPE shows **~18× lower variance** across seeds (σ = 3 vs 54).
- On simpler spaces, BO ties with Random (+0.4%), suggesting that the advantage of Bayesian optimization grows with configuration-space complexity.
- BO also discovers domain-relevant patterns automatically, such as `mtp > none > ngram` for speculative decoding and when TP=2 provides no benefit.

*Full experimental details: [`docs/09-plan-evaluacion-experimental.md`](docs/09-plan-evaluacion-experimental.md)*

## Quick start

The examples below assume that a reachable vLLM server is already running.

### Basic usage

```bash
# Install
python -m venv .venv && source .venv/bin/activate
pip install -e ".[test]"

# Single benchmark (no optimization)
python scripts/run_benchmark_once.py \
    --vllm-url http://localhost:8000 \
    --model your/model \
    --concurrency 50 \
    --num-requests 256

# Full Bayesian autotuning
python scripts/run_autotune_optuna.py \
    --vllm-url http://localhost:8000 \
    --model your/model \
    --trials 50 \
    --concurrency 16 \
    --num-requests 100
```

### Advanced usage

```bash
# Random baseline
python scripts/run_autotune_optuna.py \
    --config configs/your-config.json \
    --sampler random \
    --seed 42 \
    --ssh-host gpu.example.com \
    --ssh-user ubuntu \
    --ssh-key-path ~/.ssh/id_ed25519 \
    --auto-tunnel \
    --run-label my-random-baseline

# Dry run (validate search space without GPU time)
python scripts/run_autotune_optuna.py \
    --config configs/your-config.json \
    --dry-run

# Analyze results
python scripts/analyze_experiment.py

# Web UI
streamlit run app.py

# Tests
pytest -q
```

### Key options

| Flag | Purpose |
|---|---|
| `--sampler tpe\|random` | TPE (Bayesian, default) or Random (baseline) |
| `--config <file>` | JSON config file (search space, model, workload) |
| `--trials N` | Optimization budget (default: 50) |
| `--concurrency N` | Maximum concurrent requests per trial |
| `--num-requests N` | Prompts per trial (default: 100) |
| `--ssh-host` / `--ssh-user` | SSH connection to the GPU host |
| `--auto-tunnel` | Automatic SSH port-forward |
| `--dry-run` | Validate the search space without launching vLLM |

## Configuration space

The search space consists of **up to 24 parameters** organized in five blocks. Not all dimensions are explored simultaneously; the effective space is configured per study.

Full space definition: [`docs/05-espacio-de-configuracion.md`](docs/05-espacio-de-configuracion.md)

| Block | Key parameters |
|---|---|
| **Parallelism** | `tensor_parallel_size`, `pipeline_parallel_size`, `data_parallel_size`, `enable_expert_parallel` |
| **Memory & cache** | `max_num_seqs` (16–512), `gpu_memory_utilization` (0.70–0.97), `kv_cache_dtype` (auto/fp8), `max_model_len` (4K–262K), `enforce_eager` |
| **Scheduler** | `max_num_batched_tokens`, `enable_chunked_prefill`, `enable_prefix_caching`, `block_size`, `scheduling_policy` |
| **Speculative decoding** | `method` (none/ngram/mtp/eagle/draft_model/medusa), `num_speculative_tokens` |
| **Mamba cache** | `mamba_cache_dtype`, `mamba_ssm_cache_dtype`, `mamba_block_size`, `mamba_cache_mode` |

**Feasibility pruning:** before consuming GPU time, configurations are filtered through six checks, including structural constraints, head/layer divisibility, and an additive memory model (weights + KV + Mamba state + overhead vs. VRAM).

## Objective function

**Single-objective** (default):

```text
score = throughput_tok/s × cap_factor × floor_factor − α × latency_p95 − β × error_rate × 1000
```

- `cap_factor`: discounts low concurrency or low-context configurations.
- `floor_factor`: quadratic penalty below a usable throughput floor (default: 20 tok/s).
- `α` (default 0.01): tail-latency penalty.
- `β` (default 1.0): error-rate penalty.

**Multi-objective:** optimizes 2–7 objectives simultaneously via NSGA-II, returning a Pareto front. Available axes: throughput, context, latency, goodput, TTFT, TPOT, tokens/joule.

Full objective specification: [`docs/07-metricas-y-funcion-objetivo.md`](docs/07-metricas-y-funcion-objetivo.md)

## Prompt battery

The benchmark battery contains 794 prompts across 7 categories in `datos/prompt_banks/*.jsonl`, with one file per category:

| Category | Count | Input range | Output target |
|---|---:|---|---:|
| `simple_qa` | 111 | 6–16 tokens | 32 |
| `complex_reasoning` | 118 | 15–79 tokens | 384 |
| `code` | 118 | 11–52 tokens | 256 |
| `rag_context` | 117 | 23–98 tokens | 64 |
| `long_context` | 112 | ~24K–35K tokens | 192 |
| `mcp_tool_use` | 112 | 244–713 tokens | 512 |
| `data_processing` | 106 | 59–4.3K tokens | 384 |

Banks are generated once (seed = 42) and reused identically across trials. Six additional workload profiles are available beyond `generalist`.

Battery design: [`docs/06-diseno-bateria-prompts.md`](docs/06-diseno-bateria-prompts.md)

## Reproducibility

Every experiment produces a self-contained run directory:

```text
results/runs/<run_id>/
├── requests.jsonl          # One record per HTTP request
├── trial_<n>_manifest.json # Per-trial config + metrics
├── run_manifest.json       # Study summary + all trials
└── run.log                 # Structured log
```

Convergence plots and Pareto fronts are saved alongside. The analysis script (`scripts/analyze_experiment.py`) generates best-so-far curves, reproducibility plots, and post-hoc early-stopping estimates from these manifests.

## Documentation

> [!NOTE]
> Most in-depth design notes are currently available in Spanish.

The `docs/` directory contains the technical background and design notes behind the project, including system architecture, configuration-space design, prompt-battery construction, metrics, optimization strategy, experimental methodology, logging, SSH-based orchestration, and research extensions.

Useful entry points:
- [`docs/04-arquitectura-del-sistema.md`](docs/04-arquitectura-del-sistema.md)
- [`docs/05-espacio-de-configuracion.md`](docs/05-espacio-de-configuracion.md)
- [`docs/06-diseno-bateria-prompts.md`](docs/06-diseno-bateria-prompts.md)
- [`docs/07-metricas-y-funcion-objetivo.md`](docs/07-metricas-y-funcion-objetivo.md)
- [`docs/09-plan-evaluacion-experimental.md`](docs/09-plan-evaluacion-experimental.md)

## References

- [Snoek, Larochelle & Adams (2012), *Practical Bayesian Optimization of ML Algorithms*](https://arxiv.org/abs/1206.2944)
- [Kwon et al. (2023), *Efficient Memory Management for LLM Serving with PagedAttention*](https://arxiv.org/abs/2309.06180)
- [Bergstra et al. (2011), *Algorithms for Hyper-Parameter Optimization*](https://papers.nips.cc/paper_files/paper/2011/hash/86e8f7ab32cfd12577bc2619bc635690-Abstract.html)
- [Dalibard, Schaarschmidt & Yoneki (2017), *BOAT: Bayesian Optimization on Any Target*](https://www.cl.cam.ac.uk/~ey204/pubs/2017_WWW.pdf)

## License

[MIT](LICENSE)

vllm-bayesian-autotuner

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README