hotpin-llm

KO | EN

gitlite — search

#javascript #react #rust #deep-learning #nodejs #macos #java #awesome #cli #vue #awesome-list #pytorch #mcp #openai

(★ 8)

HotPin: routing-guided expert pinning for lossless low-RAM MoE inference on consumer CPU+NVMe - 120B model at 3.84 tok/s in 19 GB RAM, no GPU

hotpin-llm 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# hotpin-llm — HotPin: MoE Inference on Consumer CPU+NVMe

Run 30B–120B MoE models **losslessly** with less RAM than their disk size, at useful speed.

Measured on **AMD Ryzen AI 9 HX 370** (Zen5, AVX512) · 23.6 GB LPDDR5X · NVMe >1 GB/s · **CPU-only**.

[![Paper](https://img.shields.io/badge/paper-arXiv-red)](https://arxiv.org/[TO-BE-SET])
[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.21227751.svg)](https://doi.org/10.5281/zenodo.21227751)
[![License](https://img.shields.io/badge/license-AGPLv3-blue)](https://raw.githubusercontent.com/LozzKappa/hotpin-llm/main/LICENSE)
[![Commercial](https://img.shields.io/badge/commercial-contact-orange)](https://raw.githubusercontent.com/LozzKappa/hotpin-llm/main/COMMERCIAL_LICENSE.md)
[![GitHub](https://img.shields.io/badge/author-LozzKappa-black?logo=github)](https://github.com/LozzKappa)
[![X](https://img.shields.io/badge/X-LozzKappa-black?logo=x)](https://x.com/LozzKappa)
[![Reddit](https://img.shields.io/badge/Reddit-feeling__glove5995-orange?logo=reddit)](https://reddit.com/u/feeling_glove5995)
[![Discord](https://img.shields.io/badge/Discord-lozzyk__-5865F2?logo=discord)](https://discord.com/users/lozzyk_)

---

## Results

### Minimum RAM (lossless, bit-identical output)

| Model | Architecture | Disk | **Min RAM** | Savings | tok/s | IQ |
|---|---|---|---|---|---|---|
| qwen3:30b-a3b | Qwen3-MoE, 3B active | 18.0 GB | **10.4 GB** | −42% | 19.7 | 41/41 |
| gemma4:26b-a4b | Gemma4-MoE, 4B active | 16.9 GB | **10.6 GB** | −37% | 11.5 | 41/41 |
| GLM-4.7-Flash | GLM4MoeLite MLA, 3B active | 19.0 GB | **13.3 GB** | −30% | 12.4 | 40/41 |
| **gpt-oss:120b** | GPT-OSS MoE, 5B active | 58.5 GB | **19.1 GB** | **−67%** | 2.6 | 41/41 |

**Lossless** is the core guarantee: the low-RAM output is **bit-identical (SHA-256)** to a
full-RAM run — zero quality tradeoff. On a 41-question multi-domain sanity check, three of
the four models score 41/41; GLM-4.7-Flash (the smallest, 3B active) scores 40/41, missing
one multi-step temporal-reasoning question. These scores measure the base models, not HotPin
(which, being lossless, leaves them unchanged). Scoring details in the paper's Appendix B.

> A 120B model on a 23.6 GB handheld PC. A 30B model in 10.4 GB RAM. No GPU required.

### HotPin speedup (gpt-oss:120b — bigger than RAM)

| Configuration | tok/s | vs baseline |
|---|---|---|
| Baseline (mmap only) | 2.64 | — |
| L3 prefetch only | 2.87 | +8.7% |
| L1+L3 combined | 3.51 | +33.0% |
| **L1 pin only** (optimal) | **3.84** | **+45.5%** |

### HotPin on in-RAM models (flat, as the theory predicts)

| Model | L1 pin | Result |
|---|---|---|
| qwen3:30b | up to 8 GB pin | +2.1% (within noise) |
| gemma4:26b | up to 4 GB pin (3.4 GiB locked) | +0.0% to −0.1% |
| GLM-4.7-Flash | up to 4 GB pin | −0.2% to −0.6% |

**Key finding:** HotPin only helps when the model is bigger than available RAM.
For in-RAM models, the OS page cache already keeps hot experts resident — pinning adds overhead.

> **Reproducing these numbers?** See [REPRODUCIBILITY.md](REPRODUCIBILITY.md) for the exact
> model files (sources + SHA-256), decoding settings, and — importantly — which results are
> bit-exact on any machine (losslessness) vs. hardware-dependent (absolute tok/s).

---

## How It Works

MoE models activate ~3–5% of parameters per token. The rest sits cold on disk (or in page cache).
`mmap` maps the full model file into virtual address space; the OS page cache handles the rest.

```
gpt-oss:120b = 58.5 GB on disk
Working Set  = 19.1 GB in RAM   ← only pages actually read per token
Cold pages   = 39.4 GB on NVMe  ← read on-demand when a cold expert is selected
```

**HotPin L1** (`LLAMA_HOT_EXPERTS` env var): after mmap, `mlock` (Linux) /
`VirtualLock` (Windows) the top-K expert weight tensors so the OS cannot evict
them even under RAM pressure. ~40 lines of C++ in llama-model.cpp. Lossless by
construction: only page residency changes, never the weights.

**HotPin L3** (`--prefetch` flag): after each decoded token, issue async NVMe
read-ahead hints for the next token's likely experts (40–60% routing
persistence between consecutive tokens). Useful alone (+8.7%); redundant when
L1 is active (the pages are already pinned).

**AutoConfig** (`tools/autoconfig.py`): measures free RAM, applies the boundary
condition rule, and launches `llama-server` with the right hot file automatically.

### The boundary condition

```python
if disk_size > available_ram:          # AutoConfig uses a conservative 0.9 factor
    launch_with_hotpin(hot_file)       # model bigger than RAM → pinning effective
else:
    launch_mmap_only()                 # OS page cache sufficient → HotPin adds overhead
```

Validated on 4/4 tested models: +45.5% on the bigger-than-RAM side (gpt-oss:120b),
flat within noise on the in-RAM side (qwen3, gemma4, GLM).

---

## Repository Layout

```
hotpin-llm/
├── patches/                            ← apply to llama.cpp commit a6647b1 (see Quick Start)
│   ├── hotpin_l1_pin.diff              ← HotPin L1: mlock/VirtualLock expert pin (CORE)
│   ├── dump_routing_prefetch.diff      ← --dump-routing, --prefetch (L3), --batch/--raw/--chat-template
│   ├── gptoss_arch.diff                ← "gptoss" GGUF arch support (needed for gpt-oss:120b)
│   └── glm4moelite_mla_fix.diff        ← GLM-4.7-Flash MLA fix (needed for GLM only)
├── hot-files/                          ← pre-computed routing-frequency hot files
│   ├── qwen3-30b/                      ← hotR_2/4/6/8gb.txt, hot_all.txt
│   ├── gemma4-26b/                     ← hotR_2gb.txt, hotR_4gb.txt
│   ├── glm-47-flash/                   ← hotR_2gb.txt, hotR_4gb.txt
│   └── gpt-oss-120b-q4km/              ← hot_top8.txt
├── tools/
│   ├── autoconfig.py                   ← automatic launcher (boundary condition + hot file pick)
│   └── make_hot_file.py                ← routing dump → hot file (reads arch from GGUF)
├── benchmarks/                         ← the scripts behind every number in the paper
│   ├── bench_intelligence.py           ← 41-question multi-domain benchmark
│   ├── bench_l1l3.py                   ← baseline / L1 / L3 / L1+L3 speed comparison
│   ├── bench_minram.py + ballast.py    ← min-RAM lossless methodology
│   └── README.md
├── results/all_benchmarks.json         ← full measured results (JSON)
├── docs/paper/main.tex + main.pdf      ← the paper (arXiv source)
├── LICENSE                             ← GNU AGPLv3 (full text)
├── NOTICE                              ← copyright + dual-licensing summary
├── COMMERCIAL_LICENSE.md               ← commercial licensing
└── CITATION.cff                        ← academic citation
```

---

## Requirements

- **OS**: Windows 10/11 or Linux (macOS untested; L1 should work via `mlock`)
- **CPU**: AVX2 or newer (AVX512 helps)
- **RAM**: see the Min RAM column above for each model
- **Disk**: NVMe SSD strongly recommended for bigger-than-RAM models (>500 MB/s)
- **Build tools**: CMake ≥ 3.14 + a C++17 compiler (MSVC 2022 / GCC / Clang)
- **Python** ≥ 3.8 for the tools; optional packages:
  - `gguf` — lets `make_hot_file.py` read layer/expert counts from the model
    (falls back to a heuristic without it)
  - `psutil` — better free-RAM detection in `autoconfig.py` (falls back to
    OS APIs without it)

---

## Quick Start

### 1. Build llama.cpp with the HotPin patches

The patches are generated against **llama.cpp commit
`a6647b1a32f2ee456abff98ee2db15ed6e957c74`** (build ~b9857) and verified to
apply and compile cleanly there. On a different commit they may need rebasing.

```bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout a6647b1a32f2ee456abff98ee2db15ed6e957c74

# Core (always):
git apply /path/to/hotpin-llm/patches/hotpin_l1_pin.diff
git apply /path/to/hotpin-llm/patches/dump_routing_prefetch.diff

# Only if you want gpt-oss:120b (Unsloth GGUF, arch "gptoss"):
git apply /path/to/hotpin-llm/patches/gptoss_arch.diff

# Only if you want GLM-4.7-Flash (MLA fix — without it the output is garbage):
git apply /path/to/hotpin-llm/patches/glm4moelite_mla_fix.diff
```

The four patches are **orthogonal**: any subset, any order.

```bash
# Build (Linux)
cmake -B build -DLLAMA_BUILD_SERVER=ON
cmake --build build --config Release -j$(nproc)

# Build (Windows MSVC)
cmake -B build -G "Visual Studio 17 2022" -A x64 -DLLAMA_BUILD_SERVER=ON
cmake --build build --config Release --target llama-server llama-simple-chat
```

### 2. Launch with AutoConfig (recommended)

AutoConfig measures your free RAM, applies the boundary condition, picks the
largest hot file that fits, and launches `llama-server`:

```bash
python tools/autoconfig.py \
  --gguf /path/to/gpt-oss-120b-Q4_K_M-00001-of-00002.gguf \
  --hot-dir hot-files/gpt-oss-120b-q4km/ \
  --server /path/to/llama.cpp/build/bin/llama-server

# Output:
#   Model:      58.5 GB on disk        ← split GGUFs: all shards summed
#   Free RAM:   19.4 GB
#   Recommended: mmap + L1 pin (4 GB budget)
#   [A] Auto  [M] Manual  [Q] Quit
```

**What AutoConfig does, step by step:**

1. **Measures the model** — file size on disk; for split GGUFs
   (`…-00001-of-00002.gguf`) it sums all shards automatically.
2. **Measures free RAM** — via `psutil` if installed, otherwise
   `GlobalMemoryStatusEx` (Windows) / `/proc/meminfo` (Linux).
3. **Applies the boundary condition** — `disk > 0.9 × free_RAM`?
   - **Yes (bigger than RAM)** → picks the largest hot file in `--hot-dir`
     that fits the leftover budget (`free_RAM − working_set − headroom`) and
     launches `llama-server` with `LLAMA_HOT_EXPERTS=<hot file>` (**L1 pin
     only** — L3 is deliberately not added on top: measured −9.4% vs L1 alone).
   - **No (fits in RAM)** → launches plain `mmap`, no pin. Even if you pass a
     hot file, it is skipped: on in-RAM models pinning is pure overhead
     (measured on qwen3, gemma4, GLM).
4. **Interactive prompt** — `[A]` accepts the recommendation, `[M]` lets you
   set the pin budget in GB yourself, `[Q]` exits. Then the full `llama-server`
   command line is printed (so you can copy it into a script) and the server
   starts on `http://127.0.0.1:<port>` with an OpenAI-compatible API
   (`/v1/chat/completions`).

- `--server` is optional if `llama-server` is on PATH or `LLAMA_SERVER_PATH` is set.
- Other flags: `--server-port` (default 8080), `--threads`, `--ctx` (default 2048).

Once running, talk to it like any OpenAI endpoint:

```bash
curl http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":200}'
```

### 3. Manual launch

```bash
# L1 pin only (optimal for bigger-than-RAM models)
LLAMA_HOT_EXPERTS=hot-files/gpt-oss-120b-q4km/hot_top8.txt \
  llama-server -m gpt-oss-120b.gguf --mmap -t 12 -c 2048

# L3 prefetch only (when you don't want to lock RAM)
llama-server -m model.gguf --mmap --prefetch hot-files/<model>/hotR_4gb.txt -t 12

# Windows (PowerShell)
$env:LLAMA_HOT_EXPERTS = "hot-files\gpt-oss-120b-q4km\hot_top8.txt"
.\llama-server.exe -m gpt-oss-120b.gguf --mmap -t 12 -c 2048
```

On load you should see a line like:

```
load_tensors: expert-cache v1: pinned 1946 hot expert-slices (3465.2 MiB) from ...
```

### 4. Generate a hot file for a new model

```bash
# 1) one-time routing dump (5–10 varied prompts are enough)
llama-simple-chat -m model.gguf --dump-routing routing.bin -n 200 --no-color
# (type a prompt, wait for generation, repeat with different prompts, exit)

# 2) convert to a hot file sized to your RAM budget
python tools/make_hot_file.py \
  --routing routing.bin --budget-gb 4 \
  --model model.gguf --out hot-files/<model>/hotR_4gb.txt
```

`make_hot_file.py` reads `block_count`/`expert_count` from the GGUF metadata to
size expert slices correctly (`--layers/--experts` override available).
Dump format: one record per token per MoE layer —
`int32 layer, int32 k, k × int32 expert_id`.

The hot file itself is plain text, one `layer expert` pair per line, sorted by
routing frequency (most-selected experts first) — easy to inspect, diff, and share:

```
0 8
12 41
3 54
...
```

---

## Troubleshooting

| Symptom | Cause / fix |
|---|---|
| `patch does not apply` | You are not on commit `a6647b1` — run the `git checkout` from step 1. |
| No `pinned N hot expert-slices` line on load | `LLAMA_HOT_EXPERTS` not set/visible to the process, wrong path, or the model has no matching `ffn_*_exps` tensors. |
| Pinned MiB much smaller than expected | Hot file generated for a different model (layer/expert ids out of range are skipped silently). Regenerate with `make_hot_file.py` against the right GGUF. |
| `failed to apply the chat template` | The GGUF ships a Jinja template `llama-simple-chat` can't parse — pass `--chat-template <name>` (e.g. `gemma`) or `--raw`. |
| Chat outputs template markers (`<|im_start|>`…) or gibberish | You are using an **Ollama blob**: Ollama stores chat templates in its manifest, not in the GGUF, so `llama-server` falls back to ChatML. Fine for Qwen (native ChatML) and for benchmarks (raw mode); for interactive chat with other models use a HuggingFace GGUF (embedded template) or pass `--chat-template`/`--chat-template-file`. |
| GLM-4.7-Flash outputs garbage | You didn't apply `glm4moelite_mla_fix.diff`. |
| `unknown architecture: gptoss` | You didn't apply `gptoss_arch.diff`. |
| L1+L3 slower than L1 alone | Expected — prefetch is redundant once pages are pinned (−9.4% measured). Use L1 only. |
| HotPin shows ~0% gain | Your model fits in RAM — that's the boundary condition working as intended. |
| `mlock` fails / pins 0 MiB on Linux | Raise `ulimit -l` (memlock) or run with `CAP_IPC_LOCK`. |

---

## Model-Specific Notes

**gpt-oss:120b (Unsloth GGUF)** — needs `patches/gptoss_arch.diff` (the GGUF
declares arch `gptoss`, unsupported upstream at the pinned commit). Split into
2 shards: pass shard 1 to `-m`; AutoConfig sums the shards for the size check.

**GLM-4.7-Flash** — needs `patches/glm4moelite_mla_fix.diff`. Its Multi-head
Latent Attention has mismatched key dims (`key_length=256`,
`key_length_mla=576`); without the patch the output is garbage. After the
patch: 12.4 tok/s, coherent output (40/41 intelligence). Details in Appendix C of the paper.

**gemma4:26b (Unsloth UD GGUF)** — its embedded Jinja chat template is not
supported by `llama-simple-chat`; pass `--chat-template gemma`. Its fused
`ffn_gate_up_exps` tensors are handled by the L1 patch.

---

## Benchmarks

Every number above is reproducible with the scripts in [`benchmarks/`](benchmarks/):

```bash
# 41-question intelligence (temp=0, binary scoring)
python benchmarks/bench_intelligence.py --exe ./llama-simple-chat --model model.gguf

# HotPin speed comparison (4 configs)
python benchmarks/bench_l1l3.py --exe ./llama-simple-chat --model model.gguf \
    --hot-file hot-files/gpt-oss-120b-q4km/hot_top8.txt

# Min-RAM lossless (ballast + SHA-256 bit-identity check)
python benchmarks/bench_minram.py --exe ./llama-simple-chat --model model.gguf --reference ref.txt
python benchmarks/ballast.py 12   # separate terminal
python benchmarks/bench_minram.py --exe ./llama-simple-chat --model model.gguf --compare ref.txt
```

Raw results: [`results/all_benchmarks.json`](results/all_benchmarks.json).

---

## MoE vs Dense: Why This Matters

At equal disk size (~18 GB), a MoE 30B model is **9× faster** than a dense 31B on CPU:

| Model | Type | Disk | tok/s |
|---|---|---|---|
| qwen3:30b-a3b | MoE — 3B active/token | 18.0 GB | **19.7** |
| gemma4:31b-it | Dense — 31B active/token | 18.5 GB | 2.17 |

The MoE advantage is structural: only 3B parameters are computed per token instead of 31B.
HotPin then adds another +45.5% for the bigger-than-RAM case.

## CPU vs iGPU

The Radeon 890M iGPU (RDNA3.5, shared RAM) gives **−26 to −37%** vs CPU-only on this
hardware: it shares the physical RAM bus with the CPU, so offloading adds copy overhead
without adding bandwidth. GPU offload only helps with **dedicated VRAM**
(tested: GTX 1650 Ti 4 GB → +140% vs CPU).

---

## Paper

> Ibrahim Khaled. "HotPin: Routing-Guided Expert Pinning for Lossless Low-RAM
> Inference of Large MoE Models on Consumer CPU+NVMe Hardware." arXiv, 2026.

[arXiv link — coming soon] · Source: [`docs/paper/main.tex`](docs/paper/main.tex)

If you use HotPin in your research, please cite using `CITATION.cff`.

---

## License

**Open source** — [GNU AGPLv3](LICENSE) (see [NOTICE](NOTICE)).

Personal use · Academic research · Open-source (AGPLv3-compatible) · Student projects → **free**

Enterprise · SaaS · Proprietary products → network copyleft applies →
**commercial license**: ibrahimkhaled686@gmail.com — see [COMMERCIAL_LICENSE.md](COMMERCIAL_LICENSE.md).

---

## Contributing

Hot files for other MoE models are welcome — see [CONTRIBUTING.md](CONTRIBUTING.md)
(data contributions need no paperwork; code contributions require a CLA to keep
dual licensing possible). Please open a PR with:
- `hot-files/<model-slug>/hotR_Xgb.txt` (routing-frequency sorted, one `layer expert` per line)
- `results/<model-slug>.json` with hardware specs and measured tok/s

Tested MoE architectures (via patched llama.cpp):
- GPT-OSS MoE (gpt-oss:120b) — requires `gptoss_arch.diff`
- Qwen3-MoE (qwen3:30b-a3b)
- Gemma4-MoE (gemma4:26b-a4b) — fused gate_up tensors, `--chat-template gemma`
- GLM4MoeLite MLA (GLM-4.7-Flash) — requires `glm4moelite_mla_fix.diff`

hotpin-llm

hotpin-llm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

hotpin-llm

hotpin-llm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README