MiniMax-M3-2x-DGX-Spark-36-tok-s

KO | EN

gitlite — search

#javascript #python #react #typescript #llm #go #machine-learning #rust #deep-learning #nodejs #kubernetes #security #ios #mcp

MiniMax-M3-2x-DGX-Spark-36-tok-s

(★ 42)

MiniMax-M3 (428B, no pruning) at 36 tok/s on 2× NVIDIA DGX Spark — W4A16 GPTQ + NVFP4 KV + EAGLE-3 speculative decoding on vLLM. Three serving lanes: speed / balanced / long-context.

MiniMax-M3-2x-DGX-Spark-36-tok-s Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# MiniMax-M3 on 2x DGX Spark — 36 tok/s, all 428B params (no pruning)

> All 428B parameters of MiniMax-M3 (no pruning, no REAP), served at **36.6 tok/s
> JSON / 31.8 code** on two NVIDIA DGX Sparks — W4A16 GPTQ weights + 4-bit KV
> cache + EAGLE-3 speculative decoding on vLLM, TP=2 across a GB10 pair (Bluey +
> Reddie, direct 200G RoCE), 196K max context (208,128-token KV pool), single
> stream, temp 0. Verified on this pair 2026-07-05.

**This is [a3refaat](https://github.com/a3refaat)'s stack.** This repo is a
reproduction + extension of
[a3refaat/spark-vllm-docker](https://github.com/a3refaat/spark-vllm-docker)
branch `minimax-m3-4bit-w4a16` (MIT, upstream fork of
[eugr/spark-vllm-docker](https://github.com/eugr/spark-vllm-docker)), documented
in his NVIDIA forum post
[forums.developer.nvidia.com/t/375595](https://forums.developer.nvidia.com/t/375595).
The container image, the vLLM source patch, the b12x kernel integration, the
KVarN backport and its MiniMax-M3 extensions, the EAGLE3 drafter checkpoints, and
the recipes are **his work**. Our numbers **confirm his published tg32 35.49
(peak 36.79) on independent hardware**. What we add: the reproduction itself, two
build/ops fixes ([`patches/`](patches/)), the recovery of a recipe he shelved,
lever A/B results (including an honest negative), and the deployment gotchas in
[`DEFAULT-CONFIG.md`](DEFAULT-CONFIG.md). Full credits [below](#credits--links).

## TL;DR

- **What you get:** one Docker image, one model, **three serving recipes (lanes)**
  — pick by workload. More speed with less context = lane C; more context with
  less speed = lane B; **lane A is the balance** (the verified hero).
- **The numbers (lane A, verified on our pair 2026-07-05):** **36.6 tok/s JSON /
  31.8 code** at 196K max context, 208,128-token KV pool, single stream, temp 0.
  This confirms a3refaat's published **tg32 35.49 (peak 36.79)** on independent
  hardware.
- **How 428B fits:** W4A16 GPTQ (224G on disk → 112G/node at TP=2) + 4-bit KV
  cache + an EAGLE-3 drafter, on two 128G-unified GB10 boxes.
- **Who it's for:** anyone reproducing a full-size (unpruned) MiniMax-M3 on a 2x
  DGX Spark pair and wanting the exact image, recipes, and bring-up gotchas.

## Hardware

Two NVIDIA DGX Sparks (GB10, **128G unified memory each, ~121 GiB usable**),
direct-cabled **200G RoCE** on `enp1s0f0np0` / `rocep1s0f0`, `192.168.192.0/24`.

| role | node | GPU | fabric IP | notes |
| --- | --- | --- | --- | --- |
| head (rank0, serves `:8000`) | Bluey | GB10 | 192.168.192.1 | `NCCL_IB_GID_INDEX=3` |
| worker (rank1) | Reddie | GB10 | 192.168.192.2 | historically GID 5 — see [`DEFAULT-CONFIG.md`](DEFAULT-CONFIG.md) gotchas |

a3refaat's scripts use link-local ConnectX autodiscovery; ours is statically
addressed — equivalent.

**How 428B fits on two Sparks:**

- **Weights:** W4A16 GPTQ
  ([`Sebesky/MiniMax-M3-W4A16-GPTQ`](https://huggingface.co/Sebesky/MiniMax-M3-W4A16-GPTQ),
  Marlin int4 MoE) = **224G on disk → 112G per node at TP=2**, leaving only
  **~9G per node for KV cache** on a 128G-unified-memory GB10 (~121 GiB usable).
- **KV:** ~9G would be nothing at bf16 — but MiniMax-M3's sparse-attention (MSA)
  layers carry only **2 KV heads**, and both 4-bit KV stacks (b12x nvfp4, KVarN
  k4v2) compress what's left. That's how 9G/node becomes a 208,128-token pool at
  196K max context (lane A), or 262K+ drafterless (lane B).
- **Speed:** decode is weight-bandwidth-bound at this size, so EAGLE-3's
  token-amortization is the real throughput lever — the
  [`Sebesky/MiniMax-M3-EAGLE3-RTN-INT4`](https://huggingface.co/Sebesky/MiniMax-M3-EAGLE3-RTN-INT4)
  drafter (1.6G, int4-RTN incl. embed/lm_head) rides the same 4-bit KV as the
  target.

## Quick start

Byte-exact flow in [`DEFAULT-CONFIG.md`](DEFAULT-CONFIG.md). Summary:

1. Clone his branch, apply
   [`patches/dockerfile-unreachable-ref-fetch.patch`](patches/dockerfile-unreachable-ref-fetch.patch)
   (the pinned vLLM ref is unreachable on a cold cache) and
   [`patches/mod-prometheus-graceful-skip.patch`](patches/mod-prometheus-graceful-skip.patch)
   (dependency drift otherwise aborts every launch), then `./build-deploy.sh`
   (~29G image) and distribute via `docker save | ssh docker load` over the
   fabric.
2. Stage weights once: download the 224G target + 1.6G drafter on the head, rsync
   to the worker over the 200G RoCE link (~4 min vs a second 224G download).
3. `docker rm -f vllm_node` on both nodes + drop page caches, then
   `./run-recipe.sh --no-ray -d recipes/<lane>.yaml` with **GMU 0.929** (not 0.93
   — see the knife-edge gotcha).
4. Let the first request burn the ~60s b12x JIT warmup before benchmarking
   anything.

## Setup (detailed)

Full ordered flow — patch → build on head → distribute image → stage weights on
both nodes → clean containers + drop caches → serve from the head → warmup →
bench — is in [`DEFAULT-CONFIG.md`](DEFAULT-CONFIG.md). Each stage in brief:

### Weights

Prefetch on **both** nodes — serving containers run `HF_HUB_OFFLINE=1`.

- **Target:**
  [`Sebesky/MiniMax-M3-W4A16-GPTQ`](https://huggingface.co/Sebesky/MiniMax-M3-W4A16-GPTQ)
  (Marlin W4A16 MoE, **224G on disk**).
- **Drafter (lanes A/C):**
  [`Sebesky/MiniMax-M3-EAGLE3-RTN-INT4`](https://huggingface.co/Sebesky/MiniMax-M3-EAGLE3-RTN-INT4)
  (**1.6G**, int4-RTN incl. embed/lm_head). Do not substitute a bf16 or "slim"
  shared-embed drafter — the int4-everything variant is the validated one.
- Download once on the head, then `rsync` over the 200G link — **~4 minutes for
  the 224G target** vs a second Internet pull.

### Image / build

`vllm-node-minimax-m3-b12x` (~29G) = vLLM built from source at pinned ref
`979b56a66c96` + his `minimax-m3-fused-fp8-kv` source patch + Rust tool parser,
with his vendored b12x kernel library layered on top (`Dockerfile.deploy`). One
image backs all three lanes — per-recipe mods are applied into a fresh container
at launch. Clone his branch, apply the two patches, `./build-deploy.sh`, then
distribute:

```bash
docker save vllm-node-minimax-m3-b12x | ssh tonyspark2@192.168.192.2 docker load
```

### Launch

From the patched checkout on the head (`./launch-cluster.sh stop` before
switching recipes — mods must not stack):

```bash
# lane A — BALANCED, the hero config: b12x nvfp4 + EAGLE3, GMU 0.929
./run-recipe.sh --no-ray --gpu-mem 0.929 -d recipes/minimax-m3-w4a16-gptq-b12x-nvfp4-eagle3.yaml
```

Key lane-A settings (from the recipe): TP=2, `kv_cache_dtype: nvfp4`,
`attention_backend: b12x`, EAGLE3 `num_speculative_tokens: 3`,
`max_model_len: 196608`, `max_num_seqs: 1`, `block_size: 128`. Serve with **stock
NCCL** — do NOT export `NCCL_PROTO=LL` / `NCCL_MAX_NCHANNELS=2` (see Benchmarks).

### Verify

Wait for `http://192.168.192.1:8000/health` → 200 (model load + Triton warmup is
several minutes), then smoke-test:

```bash
curl http://192.168.192.1:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Sebesky/MiniMax-M3-W4A16-GPTQ",
  "messages": [{"role": "user", "content": "Reply exactly: OK M3 LIVE"}], "max_tokens": 16}'
```

**Never benchmark the first request after boot** — it pays the ~60s b12x JIT
warmup. Burn one throwaway request per workload shape first.

## Benchmarks

Lane A verified 2026-07-05 on our pair (Bluey + Reddie). **Lanes B and C are
currently under test** — their reference numbers below are the author's,
measured on his own 2x-GB10 pair. Protocol (ours, same discipline as the MiMo
repos): 256-token probes, temperature 0, single stream, warm engine (never the
first request after boot), `completion_tokens / wall`. Full numbers, protocol,
and boot evidence: [`benchmarks/RESULTS.md`](benchmarks/RESULTS.md).

| lane | config | our decode t/s (2026-07-05) | author's decode t/s | max ctx / KV pool | status |
| --- | --- | --- | --- | --- | --- |
| **A — BALANCED** | b12x nvfp4 KV + EAGLE3 spec-3, GMU 0.929 | **36.6 JSON / 31.8 code** (thinking disabled) | tg32 35.49, peak 36.79 | 196K / **208,128 tokens** | ✅ verified, serving now |
| **C — SPEED** | KVarN k4v2 + EAGLE3 | ⏳ currently testing | 39.7 / 30.4 / 26.4 @512/65K/120K (accept len 3.36, 78.6% @120K) | 131K / 170,752 | ⏳ testing on our pair |
| **B — CONTEXT** | KVarN k4v2, drafterless | ⏳ currently testing | ~22.7, flat with depth | 262K (370K observed) | ⏳ testing on our pair |

All single-stream, `max_num_seqs: 1`. Lane A measured on our pair (256-token
probes, temp 0, single stream, warm engine, GMU 0.929):

| workload | thinking disabled | adaptive |
| --- | ---: | ---: |
| JSON | **36.6** | 35.4 |
| code | 31.8 | 29.2 |

This reproduces a3refaat's published tg32 35.49 (peak 36.79) on independent
hardware.

**Levers tested (lane A, A/B at identical config):**

- **Thinking off beats adaptive:** +1.2 tok/s JSON (36.6 vs 35.4), +2.6 code
  (31.8 vs 29.2). Acceptance engineering works on M3, as it did on our MiMo
  DFlash deploys.
- **`NCCL_PROTO=LL` + `NCCL_MAX_NCHANNELS=2` — LOSES 5–9% on M3** (code 28.8 vs
  31.8, JSON ~33.8–35.4 vs 36.6), despite gaining real speed on a 32B-class
  dense-stream model on this same fabric (our
  [MiMo DFlash repos](https://github.com/tonyd2wild/MiMo-V2.5-DFlash-NVFP4-KV-2x-DGX-Spark)).
  Hypothesis: 428B MoE all-reduces are bandwidth-heavy and LL favors tiny
  messages. **Takeaway: NCCL lever choice is model-class-dependent — A/B it per
  deployment, don't copy it across model sizes.**

## Configuration

**Pick a lane by workload.** One image, one model, three recipes:

| lane | recipe | KV / drafter | max ctx / KV pool | best for |
| --- | --- | --- | --- | --- |
| **A — BALANCED** (hero) | [`b12x-nvfp4-eagle3`](recipes/minimax-m3-w4a16-gptq-b12x-nvfp4-eagle3.yaml) | b12x nvfp4 KV + EAGLE3 spec-3 | 196K / 208,128 tokens | the default balance |
| **C — SPEED** | [`kvarn-eagle3`](recipes/minimax-m3-w4a16-gptq-kvarn-eagle3.yaml) (his shelved recipe, restored from git history) | KVarN k4v2 + EAGLE3 | 131K / 170,752 tokens | max throughput, less context |
| **B — CONTEXT** | [`kvarn`](recipes/minimax-m3-w4a16-gptq-kvarn.yaml) | KVarN k4v2, no drafter | 262K (370K observed) | max context, less speed |

Provenance of every yaml (all reproduced byte-exact from his branch; lane C
recovered from his git history) is in [`recipes/README.md`](recipes/README.md). A
fourth recipe,
[`b12x-fp8-eagle3`](recipes/minimax-m3-w4a16-gptq-b12x-fp8-eagle3.yaml), is his
fp8-KV production config — included byte-exact, not a headline lane here.

**Tunable knobs (lane A defaults):**

- **`gpu_memory_utilization`:** **0.929 on this pair** (his recipe default is
  0.93). 0.93 failed the startup free-memory check by 0.02 GiB on our boot state;
  0.925 boots but shrinks KV below the 196K requirement; 0.929 boots with ~100MB
  margin and a 208,128-token pool. Bracket it on your own hardware — the failure
  modes are clean (too-high reports the exact GiB shortfall, too-low reports the
  exact pool size).
- **`max_model_len`:** 196608 (lane A). Lanes trade context for speed — 131K (C),
  262K+ (B).
- **`kv_cache_dtype`:** `nvfp4` (lane A) or KVarN `k4v2` (lanes B/C) — both 4-bit.
- **`num_speculative_tokens`:** 3 (EAGLE3 spec-decode; lanes A/C). Lane B is
  drafterless.
- **`max_num_seqs`:** 1 — all published numbers (his and ours) are single-
  sequence; concurrency against the 208K pool is untested.
- **`thinking_mode`:** his recipes default the chat template to adaptive/enabled;
  disable per request via `chat_template_kwargs` for throughput-critical
  structured workloads (+1.2 JSON / +2.6 code — see Benchmarks).
- **NCCL:** stock — do not export `NCCL_PROTO=LL` / `NCCL_MAX_NCHANNELS=2`
  (loses 5–9% on M3; see Benchmarks).

## Troubleshooting

All hit live on this pair; exact error signatures and fixes in
[`DEFAULT-CONFIG.md`](DEFAULT-CONFIG.md):

1. **Cold-cache build death** — the pinned vLLM ref `979b56a66c96` lives only on
   a deleted branch (`refs/pull/45381/head` reaches it); patched at 3 fetch sites
   ([patch 1](patches/dockerfile-unreachable-ref-fetch.patch)).
2. **Dependency drift aborts the launch** — prometheus-fastapi-instrumentator
   8.0.2 already ships the fix his routing mod applies → anchor mismatch → whole
   launch aborts. The mod now detects `_resolve_path` and skips gracefully
   ([patch 2](patches/mod-prometheus-graceful-skip.patch)).
3. **Stale placeholder containers** from aborted launches eat the next launch's
   exec-script (`/workspace/exec-script.sh: No such file`) — `docker rm -f
   vllm_node` on BOTH nodes before every retry.
4. **GMU knife-edge, bracketed precisely:** 0.93 failed by 0.02 GiB on our boot
   state; 0.925 boots but shrinks KV below the 196K requirement; **0.929 is the
   sweet spot** (boots with ~100MB margin, pool 208,128).
5. **Per-node usernames differ** (tonyspark1/tonyspark2) — fine, IF the head has
   a `~/.ssh/config` Host entry mapping the worker IP to its user.
6. **Never benchmark the first request** — it pays the b12x JIT warmup (~60s).
7. **Weight staging** — download once, rsync over the fabric (~4 min vs a second
   224G download).
8. **Reddie fabric IP can vanish after teardown** — if `192.168.192.2` is
   missing, `sudo ip addr add 192.168.192.2/24 dev enp1s0f0np0`.
9. **Bluey weight-load UVM thrash** when free RAM < 2G during the 224G load — run
   a cache-dropper loop during load, but never once the load path is under
   `instanttensor`.

## Credits & links

This deployment stands almost entirely on prior public work:

- **a3refaat** — the whole deployment package this repo reproduces:
  [a3refaat/spark-vllm-docker](https://github.com/a3refaat/spark-vllm-docker)
  branch `minimax-m3-4bit-w4a16` — the b12x/KVarN MiniMax-M3 integration mods, the
  fused fp8-KV vLLM patch, the recipes, the GB10 unified-memory operational rules,
  and the forum write-up
  ([forums.developer.nvidia.com/t/375595](https://forums.developer.nvidia.com/t/375595)).
  Our lane-A numbers are a confirmation of his, on hardware he's never touched.
- **Sebesky** — the HF checkpoints this deploy loads:
  [`Sebesky/MiniMax-M3-W4A16-GPTQ`](https://huggingface.co/Sebesky/MiniMax-M3-W4A16-GPTQ)
  (W4A16 GPTQ target) and
  [`Sebesky/MiniMax-M3-EAGLE3-RTN-INT4`](https://huggingface.co/Sebesky/MiniMax-M3-EAGLE3-RTN-INT4)
  (int4-RTN EAGLE3 drafter).
- **Inferact** — the original MiniMax-M3 EAGLE3 drafter the int4 checkpoint
  derives from.
- **huawei-csl / philippebich** — KVarN calibration-free KV-cache quantization and
  its vLLM implementation
  ([PR #46812](https://github.com/vllm-project/vllm/pull/46812)).
- **gpieceoffice** — pioneered KVarN on DGX Spark
  ([forums.developer.nvidia.com/t/375421](https://forums.developer.nvidia.com/t/375421)).
- **eugr (Eugene Rakhmatulin)** — the upstream
  [spark-vllm-docker](https://github.com/eugr/spark-vllm-docker) cluster tooling
  a3refaat's branch builds on (MIT).
- **MiniMax** — the MiniMax-M3 model. Note the weights are under the **MiniMax
  Community License** — non-commercial constraints apply to the model itself,
  independent of any code license in this repo.
- **The vLLM project** — the engine everything runs on.

### License

- **Our original content** (README, DEFAULT-CONFIG, benchmark docs):
  [Apache-2.0](LICENSE).
- **The reproduced recipes** (`recipes/*.yaml`) and the `patches/` diffs against
  his MIT-licensed files remain under MIT with the upstream notice preserved:
  [LICENSE.upstream-MIT](LICENSE.upstream-MIT). Recipe comment headers are kept
  byte-exact.
- **Model weights are separate**: MiniMax-M3 is under the MiniMax Community
  License (non-commercial constraints); the GPTQ/EAGLE3 checkpoints, base images,
  and NVIDIA tooling carry their own upstream terms.

MiniMax-M3-2x-DGX-Spark-36-tok-s

MiniMax-M3-2x-DGX-Spark-36-tok-s Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

MiniMax-M3-2x-DGX-Spark-36-tok-s

MiniMax-M3-2x-DGX-Spark-36-tok-s Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README