qwopus-dflash

KO | EN

gitlite — search

#monitoring #database #dotnet #markdown #llm #json #artificial-intelligence #sql #reactjs #self-hosted

(★ 9)

DFlash speculative decoding for Qwopus3.5-9B-Coder on upstream llama.cpp

File Explorer

ZIP Download

- patch-draft-config.py
- serve.sh
.gitignore
convert-draft.sh
demo.gif
README.md
run.sh

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Up to 4.9x faster code generation from [Qwopus3.5-9B-Coder](https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder) on an RTX 3070, same outputs. DFlash block-diffusion speculative decoding on upstream llama.cpp.

![side by side: baseline vs dflash applying the same edit to a Go file](https://raw.githubusercontent.com/Gaurav-Gosain/qwopus-dflash/main/demo.gif)

62 to 304 tok/s renaming a struct field across a Go file. Both panes are real captured token streams: same model, same prompt, temperature 0. The right pane just has a DFlash draft attached.

**Draft GGUFs: [GauravGosain/Qwopus3.5-9B-Coder-DFlash-GGUF](https://huggingface.co/GauravGosain/Qwopus3.5-9B-Coder-DFlash-GGUF)**

## Quick start

```sh
llama-server \
  -hf Jackrong/Qwopus3.5-9B-Coder-GGUF:Q3_K_M \
  -hfd GauravGosain/Qwopus3.5-9B-Coder-DFlash-GGUF:Q4_K_M \
  --no-mmproj \
  --spec-type draft-dflash --spec-draft-n-max 15 \
  -fa on --jinja -c 4096 -ctk q8_0 -ctv q8_0 -ctxcp 2 -fitt 256
```

Downloads both models from Hugging Face on first run. Needs llama.cpp master (DFlash landed in [#22105](https://github.com/ggml-org/llama.cpp/pull/22105)). Or clone this repo: `./convert-draft.sh` builds the draft from scratch, `./run.sh` serves it from local files.

Three things learned the hard way: pass `-hf` and `-hfd` together (a local `-m` combined with `-hfd` currently fails to resolve the draft path); keep `--no-mmproj` (the target repo ships a 921 MB vision projector you do not need); and know that the automatic VRAM fit cannot pre-measure a DFlash draft, so with barely-enough VRAM it fails to start instead of spilling layers. Set `-ngl` yourself in that case.

## Why this repo

No DFlash draft exists for Qwopus, only for its base model ([z-lab/Qwen3.5-9B-DFlash](https://huggingface.co/z-lab/Qwen3.5-9B-DFlash)). The published GGUF conversions of that draft are not a drop-in fit:

- Qwopus extends the Qwen3.5 tokenizer with 7 extra tokens (ids 248070 to 248076), so drafts converted against the base tokenizer fail the vocab compatibility check or mistokenize.
- Some published conversions target a llama.cpp fork rather than upstream.

The fix is cheap because of how DFlash works in llama.cpp: the draft GGUF carries no token embeddings and no lm_head, it borrows the target model's copies at runtime. Converting a Qwopus-matched draft therefore needs only the Qwopus tokenizer, not its 18 GB of weights, and the resulting draft matches the target embeddings exactly by construction.

## Benchmarks

RTX 3070 8 GB, target Qwopus3.5-9B-Coder Q3_K_M, draft Q4_K_M, 600-token coding generation at temperature 0, llama.cpp master (4193ea697), measured back to back with both configurations fully on GPU.

| workload | baseline | DFlash | speedup | acceptance |
| --- | --- | --- | --- | --- |
| code editing (rename a field, echo the file) | 62 tok/s | 304 tok/s | 4.9x | 0.84 |
| fresh code generation | 58 tok/s | 145 tok/s | 2.5x | 0.34 |

The speedup tracks how predictable the output is. Editing existing code is DFlash's best case: mean draft length hits 13.7 of 15, so most blocks verify in one pass. Fresh generation still runs 2.5x. Freeform prose drops to about 0.15 acceptance, still a net win. Shorter blocks raise acceptance but lower throughput (n-max 7: 0.56 acceptance, slower overall). Draft quantization barely matters (Q8_0 within noise of Q4_K_M) so the smaller Q4_K_M is the default.

Caveat: this needs about 6.5 GB of free VRAM and a low `--fit-target` margin (run.sh uses `-fitt 256`; the default 1024 reserves too much and spills layers). If other processes hold VRAM, llama-server spills target layers to CPU and speculation goes net-negative: measured 28 tok/s against a 58 tok/s baseline with the GPU shared. Do not pin `-ngl`; the automatic fit degrades gracefully instead of failing to start.

## Modes

`Q4=1 ./run.sh` serves the Q4_K_M target instead of Q3_K_M (better quality, some CPU spill on 8 GB).

## Apple Silicon (MLX)

The same pairing works on Macs via [dflash-mlx](https://github.com/bstnxbt/dflash-mlx). Measured on an M3 Pro 18 GB with `snsnc/Qwopus3.5-9B-Coder-MLX-4bit` as target:

| workload | baseline | DFlash | speedup | acceptance |
| --- | --- | --- | --- | --- |
| code editing | 28.3 tok/s | 55.3 tok/s | 1.95x | 0.83 |
| fresh code generation | 28.4 tok/s | 42.3 tok/s | 1.49x | 0.66 |

The 4-bit 9B decodes at the unified-memory bandwidth roofline (about 29 tok/s on 150 GB/s measured), so speculative verification is the only way past it. The smaller multiplier versus CUDA is hardware: a batch-16 verify pass costs 3.8x a single-token step on the M3 Pro versus roughly 1.05x on a 3070, which caps any implementation near 3.5x. Two tuning results that matter: quantize the draft (`--draft-quant w4`, the bf16 draft costs more bandwidth than its extra acceptance is worth, and peak memory drops from 7.9 to 6.1 GB) and cap verify blocks at 8 (`--block-tokens 8`, verification cost grows super-linearly on the hybrid-attention kernels, so shorter blocks net more).

Setup and serve: [`mlx/serve.sh`](mlx/serve.sh). One wrinkle: z-lab's draft config uses transformers-v5 nested keys that dflash-mlx does not read yet; [`mlx/patch-draft-config.py`](mlx/patch-draft-config.py) downloads the draft and flattens `rope_theta` and `block_size`.

## Notes and gotchas

- Qwopus ships a transformers v5 `tokenizer_config.json` (`tokenizer_class: TokenizersBackend`) that transformers v4 refuses to load. The conversion script rewrites it to `PreTrainedTokenizerFast`; the tokenizer.json itself is a standard fast tokenizer.
- Qwen3.5 is a hybrid linear-attention architecture. Every context checkpoint stores the full recurrent state (roughly 100 MB), and llama-server defaults to 32 checkpoints per slot, which alone is 3.2 GB. `run.sh` caps this with `-ctxcp 2`.
- The DFlash mask token id is 248077, which sits just past Qwopus's last added token (248076). No collision, but worth knowing if you retarget another finetune.
- `--spec-draft-n-max` is clamped to block_size - 1 = 15. Long blocks win on code even though per-token acceptance drops, because each verify pass lands about 6 tokens.

qwopus-dflash

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

qwopus-dflash

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README