llm-emu

KO | EN

gitlite — search

#python #hacktoberfest #ai #machine-learning #docker #macos #java #windows #awesome #cli #vue #pytorch #openai #open-source

(★ 10)

vLLM wall-clock emulation pluging to allow running the real vllm serve and benchmark without gpu with simple mocking

llm-emu 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM-Emu

Profile-driven online emulator for [vLLM](https://github.com/vllm-project/vllm).
LLM-Emu replaces vLLM's GPU forward pass with a latency draw from a
profile pack captured once on a real GPU. The scheduler, HTTP stack,
admission path, tokenizer, and output pipeline are vLLM's own
unmodified code; only the per-step latency is faked.

This repository accompanies the paper *LLM-Emu: Profile-Driven
Online Emulation of LLM Serving Systems* (anonymous submission).

The repository contains **only the LLM-Emu-specific code**: the
plugin (`vllm_emulator/`), the reproduce-the-paper scripts (`tools/`),
and a small **vLLM patch** under `vllm_patches/` that wires the plugin
into `vllm.v1.engine.core`, `vllm.v1.executor.uniproc_executor`, and
`vllm.v1.core.sched.scheduler`. Everything else is upstream vLLM at
v0.18.1.

---

## What's in this repo

```
vllm_emulator/                    LLM-Emu plugin (~1.7K LoC online +
                                  ~0.6K LoC offline tooling, no upstream
                                  vLLM code)
  oracle/                         density-aware step-latency oracle
                                  (adaptive-K Shepard pooling)
  hooks/executor_hook.py          intercepts execute_model(); returns a
                                  pending Future whose timer resolves
                                  after the predicted step latency
  platform.py                     vLLM platform-plugin entry point
  cuda_mock.py                    torch.cuda stubs for CPU-only host mode
  profile/build_serving_profile_filtered.py
                                  trace -> profile-pack builder
  profile/{loader,validator}.py   pack load + schema check
  profiler/trace_profiler.py      StepCycleTracer for real-GPU profile
                                  capture

tools/                            14 reproduce-the-paper scripts (see
                                  docs/reproduce.md)

vllm_patches/                     vLLM v0.18.1 in-tree wiring (~170 lines)
  llm-emu-vllm-0.18.1.patch    unified diff vs upstream
  overrides/                      pre-patched copies of the 3 changed
                                  files, mirroring vLLM's path layout
  BASE_COMMIT                     upstream commit hash + version pin

docs/reproduce.md                 step-by-step paper Table 1 reproduce
```

---

## Install (two paths)

### Path A — apply the patch (recommended)

If you have a vLLM source checkout (or you `pip install vllm==0.18.1
--no-binary vllm` and want to patch it):

```bash
pip install vllm==0.18.1
git clone <this-repo> llm-emu && cd llm-emu

# Patch the installed vLLM (or your local clone of vllm-project/vllm
# at commit a26e8dc7f0):
VLLM_DIR=$(python -c "import vllm, os; print(os.path.dirname(vllm.__file__))")
cd "$VLLM_DIR/.." && patch -p1 < /path/to/llm-emu/vllm_patches/llm-emu-vllm-0.18.1.patch

# Install the plugin
cd /path/to/llm-emu && pip install -e .
```

If plan to extend to other versions of vllm, just rebase this over the vllm target versions and address the potential conflication. 
Since the vllm changes is only appending new flags,  exposing internal status and wiring to the emulator mocker if enabled without modifying any vllm logics/functionality, 
the risk of major conflication is consider to be small.

### Path B — drop in the override files

If you can't or don't want to apply the patch (e.g., a compiled wheel):

```bash
pip install vllm==0.18.1
git clone <this-repo> llm-emu && cd llm-emu

VLLM_DIR=$(python -c "import vllm, os; print(os.path.dirname(vllm.__file__))")

# Replace the three changed vLLM files with the pre-patched copies
cp vllm_patches/overrides/vllm/v1/engine/core.py            "$VLLM_DIR/v1/engine/core.py"
cp vllm_patches/overrides/vllm/v1/executor/uniproc_executor.py  "$VLLM_DIR/v1/executor/uniproc_executor.py"
cp vllm_patches/overrides/vllm/v1/core/sched/scheduler.py   "$VLLM_DIR/v1/core/sched/scheduler.py"

# Install the plugin
pip install -e .
```

Both paths register the `vllm.platform_plugins` entry point
`emulator -> vllm_emulator.platform:emulator_platform_plugin`. With
`VLLM_EMULATOR_ENABLE_ORACLE` unset, the plugin is a no-op and vLLM
runs as usual.

---

## CLI usage

The plugin is selected by env var on the same `vllm serve` line you'd
use without it:

```bash
# Real vLLM (no emulator):
vllm serve Qwen/Qwen3-8B \
  --port 8000 --max-model-len 4096 --gpu-memory-utilization 0.85

# LLM-Emu emulator (no GPU forward pass; uses profile pack):
VLLM_EMULATOR_ENABLE_ORACLE=1 \
VLLM_EMULATOR_PROFILE_PACK=path/to/serving-full.json \
# Disable GPU for safeguarding
CUDA_VISIBLE_DEVICES="" \
vllm serve Qwen/Qwen3-8B \
  --port 8001 --max-model-len 4096 --gpu-memory-utilization 0.85
```

The profile pack is generated by `tools/adaptive_profile_capture.sh`;
see `docs/reproduce.md`. For a quick start without capturing your own,
we ship a ready-to-use pack at
`example_profiling_data/A40-Q8-Qwen3-8B.json` (Qwen3-8B on A40, the
A40-Q8 row of the table below).

---

## Reproduce Results

We got this results when running the emulator on Qwen models and RTX8000/A40 GPU hosts as shown in the paper

| Metric | $r=2$ | $r=4$ | $r=8$ | $r=16$ | $r=32$ |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Main: Qwen3-8B at RTX 8000** | | | | | |
| TTFT | -9.81% | -6.92% | -1.89% | +3.17% | +2.52% |
| TPOT | +0.48% | +0.85% | +0.18% | +0.08% | +0.80% |
| ITL | +0.43% | +0.80% | -0.00% | -0.03% | +0.71% |
| E2E | +0.34% | +0.72% | -0.42% | +2.01% | +2.02% |
| TPS | -0.00% | -0.02% | +0.67% | +0.12% | -0.78% |
| **Model-scale up: Qwen3-14B at RTX 8000** | | | | | |
| TTFT | -9.10% | -4.42% | +1.39% | +1.24% | +1.80% |
| TPOT | +0.47% | +1.37% | -1.02% | +0.27% | +0.58% |
| ITL | +0.42% | +1.28% | -1.33% | +0.39% | +0.36% |
| E2E | +1.06% | +1.50% | +0.48% | +1.09% | +1.51% |
| TPS | +0.54% | -0.26% | +1.10% | -0.27% | -0.02% |
| **Main (Hardware swap): Qwen3-8B at A40** | | | | | |
| TTFT | -7.37% | -9.22% | -1.06% | +5.05% | +3.44% |
| TPOT | +0.94% | +1.05% | -0.46% | +1.24% | +1.52% |
| ITL | +0.82% | +1.03% | -0.62% | +1.26% | +1.25% |
| E2E | +0.74% | +0.92% | -0.73% | +3.72% | +2.83% |
| TPS | -0.02% | +0.02% | +1.25% | -1.10% | -1.03% |
| **Bursty workload: $\gamma=0.25$** | | | | | |
| TTFT | -6.90% | +4.97% | **-10.41%** | -3.13% | -2.45% |
| TPOT | +0.08% | **+4.75%** | -2.39% | -0.90% | -1.60% |
| ITL | -0.14% | **+4.69%** | -2.34% | -0.93% | -1.26% |
| E2E | -0.20% | +4.77% | -4.09% | -2.32% | -2.12% |
| TPS | -0.47% | -0.71% | **-1.88%** | +0.04% | +1.01% |
| **Model-family swap: Llama-3.1-8B at A40** | | | | | |
| TTFT | -6.52% | -9.32% | -2.30% | +4.38% | +2.12% |
| TPOT | +0.65% | +0.46% | +0.30% | +1.98% | +0.73% |
| ITL | +0.57% | +0.38% | +0.36% | +1.92% | +0.61% |
| E2E | +0.50% | +0.28% | -0.09% | +3.48% | +1.69% |
| TPS | -0.01% | -0.04% | -0.75% | -1.76% | -0.45% |
| **Model-scale down: Qwen3-4B at A40** | | | | | |
| TTFT | -6.88% | -5.16% | -3.10% | +7.78% | +4.80% |
| TPOT | +0.34% | +1.28% | +3.10% | +1.70% | +1.13% |
| ITL | +0.24% | +1.21% | +3.01% | +1.68% | +0.79% |
| E2E | +0.16% | +1.14% | +3.02% | **+5.22%** | +3.60% |
| TPS | -0.00% | +0.01% | -0.23% | -1.50% | -0.66% |

See **[docs/reproduce.md](docs/reproduce.md)** for the per-cell
commands (six cells: M-Q8, M-Q14, A40-Q8, M-Q8-Burst, A40-L8, A40-Q4).

Quick start (M-Q8 cell, ~4-5 hours wall on RTX 8000):

```bash
python tools/download_filter_sharegpt.py     # ~5 min, fetches ShareGPT
mkdir -p $HOME/cuda_stubs && cd $HOME/cuda_stubs
ln -sf /usr/local/cuda/targets/x86_64-linux/lib/stubs/libcuda.so libcuda.so.1
ln -sf /usr/local/cuda/lib64/libcudart.so.12 .
cd -

CELL_TAG=mq8-reproduce \
BENCH_MODEL=Qwen/Qwen3-8B \
HW_PREFIX=RTX-8000 \
STUB_DIR=$HOME/cuda_stubs \
bash tools/_orch_3stage_detkv.sh

cat results/mq8-reproduce/per_rate_deltas.csv
```

---

## Environment variables

| Var | Purpose |
| --- | --- |
| `VLLM_EMULATOR_ENABLE_ORACLE` | `1` activates the plugin; unset = pass-through |
| `VLLM_EMULATOR_PROFILE_PACK`  | path to `serving-full.json` |
| `VLLM_EMULATOR_MODE`          | `realtime` (sleep for predicted latency) or `accelerated` (no sleep, still not full tested) |
| `VLLM_EMULATOR_MOCK_CUDA`     | `1` activates `torch.cuda` stubs if only CPU allowed, enabled by default|

---

## License and Cititation

Apache-2.0 (inherits from upstream vLLM).

If you find this tools is useful, please consider to cite as:

```
@misc{da2026llmemunativeruntimeemulation,
      title={LLM-Emu: Native Runtime Emulation of LLM Inference via Profile-Driven Sampling}, 
      author={Wei Da and Evangelia Kalyvianaki},
      year={2026},
      eprint={2605.00616},
      archivePrefix={arXiv},
      primaryClass={cs.DC},
      url={https://arxiv.org/abs/2605.00616}, 
}
```

llm-emu

llm-emu 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

llm-emu

llm-emu 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README