ds4-dfm-rs

KO | EN

gitlite — search

#claude #nodejs #privacy #rag #python #api #open-source #cloud-native #html #list

(★ 9)

Rust-host continuation of DwarfStar for explicit, hardware-validated large-model inference on DGX-Spark.

File Explorer

ZIP Download

- - host-parity.yml
- FUNDING.yml
- - build.rs
  - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - Cargo.toml
- - common.cuh
  - ds4_fattn.cu
  - ds4_ggml_stubs.cu
  - ds4_ggml_stubs.h
  - ds4_mmq.cu
  - ds4_mmq.h
  - ds4_mmq_d2r.cu
  - ds4_mmq_d2r.cuh
  - ds4_mmq_pipe.cuh
  - ds4_repack.cu
  - ds4_repack.h
  - ggml-common.h
  - ggml-cuda.h
  - ggml-impl.h
  - ggml.h
  - inkling_attention.cuh
  - inkling_mmvq.cuh
  - inkling_q3.cuh
  - inkling_q4.cuh
  - inkling_shared_tile.cuh
  - mma.cuh
  - mmid.cu
  - mmid.cuh
  - mmq.cuh
  - mmvq.cu
  - mmvq.cuh
  - quantize.cu
  - quantize.cuh
  - unary.cuh
  - vecdotq.cuh
  - VENDOR.md
- qwen38_ple.cu
- qwen38_ple.h
- step37_primitives.cuh
- step37_vision.cuh
- - eval_prompts.txt
  - succinct.txt
  - verbose.txt
- - verbosity.f32
  - verbosity.json
- - build_direction.py
  - run_sweep.py
- .gitignore
- README.md
- Dockerfile
- - plot-qwen-ple-fp8.py
  - qwen-image-agent-2026-09-07.csv
  - qwen-image-text-2026-09-07.csv
  - qwen-images-2026-09-07.csv
- - v0.1.0.md
  - v0.1.1-receipts.json
  - v0.1.1.md
  - v0.1.2-receipts.json
  - v0.1.2.md
- - ARCHITECTURE.md
  - BASELINE.md
  - ENGINE_GAPS.md
  - FFI_CONTRACT.md
  - PARITY_MATRIX.md
  - QWEN_V065_RESTAMP_2026-08-31.md
  - README.md
  - SPLIT_READINESS.md
- chat-templates.md
- dots3-optimization-2026-09-06.md
- ds4-api-surface-matrix.md
- ds4-dfm-model-families.md
- ds4-perf.md
- ds4-w2a16-fused-dequant-gemm-design.md
- host-memory-guard.md
- inkling-optimization-2026-09-10.md
- inkling-optimization-2026-09-11-r16.md
- inkling-optimization-2026-09-11.md
- inkling-optimization-2026-09-12-r22.md
- inkling-optimization-2026-09-12-r25.md
- inkling-optimization-2026-09-12.md
- inkling-small.md
- k2-optimization-2026-09-05-cont.md
- k2-optimization-2026-09-05.md
- LINEAGE.md
- motif3-partial-reuse-2026-08-22.md
- prefill-decode-optimization-playbook.md
- qwen38-image-2026-09-07.md
- qwen38-image-input-spec.md
- qwen38-long-context-prefill-2026-09-04.md
- qwen38-long-context-throughput.png
- qwen38-long-context-throughput.svg
- qwen38-perf-2026-09-14.md
- qwen38-ple-fp8-base.png
- qwen38-ple-fp8-uncensored.png
- qwen38-ple-fp8.md
- qwen38-prefill-2026-09-06.md
- qwen38-prefill-2026-09-07.md
- README.md
- solar-open2-2026-09-07-baseline-sweep.csv
- solar-open2-2026-09-07-rounds.csv
- solar-open2-2026-09-07-throughput.png
- solar-open2-2026-09-12-evidence.json
- solar-open2-2026-09-12-r5-evidence.json
- solar-open2-2026-09-12-r5-rounds.csv
- solar-open2-2026-09-12-rounds.csv
- solar-open2-optimization-2026-09-07.md
- solar-open2-optimization-2026-09-12-r5.md
- solar-open2-optimization-2026-09-12.md
- solar-partial-reuse-2026-08-21.md
- step37-initial.md
- step37-optimization-2026-09-13-r2.md
- step37-optimization-2026-09-13-r3.json
- step37-optimization-2026-09-13-r3.md
- step37-optimization-2026-09-13.md
- step37-serving-2026-09-13.md
- - README.md
- - README.md
  - splice_mixed_expert_layers_gguf.py
- - collect_official.py
  - compare_scores.py
  - prompts.jsonl
  - README.md
  - score_official.c
- .gitignore
- deepseek4-quantize.c
- dspark_extract.py
- dspark_probe.py
- Makefile
- quants.c
- quants.h
- README.md
- argsort.metal
- bin.metal
- concat.metal
- cpy.metal
- dense.metal
- dsv4_hc.metal
- dsv4_kv.metal
- dsv4_misc.metal
- dsv4_rope.metal
- flash_attn.metal
- get_rows.metal
- glu.metal
- moe.metal
- norm.metal
- repeat.metal
- set_rows.metal
- softmax.metal
- sum_rows.metal
- unary.metal
- - README.md
- - README.md
- ANTHROPIC_LIVE_CONTINUATION.md
- COMPACT.md
- cuda-env-vars.md
- RESPONSE_API.md
- - ds4_bridge.c
  - ds4_bridge.h
  - ds4_host_load.h
- - gb10_exact_mtp.csv
  - gb10_nomtp.csv
  - gb10_nomtp_vs_exact_mtp_ts.svg
  - README.md
- .gitignore
- abba_restamp.sh
- abort_paths_gate.sh
- admission_burst_gate.sh
- admission_credit_gate.sh
- bank_churn_soak.sh
- bank_mutation_gate.sh
- bank_persist_gate.sh
- bankplan_gate.sh
- completion_stream_gate.sh
- cont_promotion_gate.sh
- cont_registry_gate.sh
- cont_streaming_gate.sh
- credit_tranche_gate.sh
- ctx_default_gate.sh
- d0a_census_gate.sh
- d0b_shadow_gate.sh
- d1a1_source_gate.sh
- d1a_residency_gate.sh
- d33_getenv_gate.sh
- d3_close_gate.sh
- deep_admit_ab_gate.sh
- deep_capacity_leg.sh
- deep_capacity_limit.sh
- deep_ctx_gate.sh
- deep_decode_profile.sh
- drafter_disarm_gate.sh
- emit_keep_gate.sh
- error_envelope_gate.sh
- exl3_stamps.sh
- finish_reason_gate.sh
- full1m_gate.sh
- gb10.csv
- gb10_spark.csv
- gb10_spark_ts.svg
- inc3_close_gate.sh
- inc3_perf_abba.sh
- inc4_close_gate.sh
- inc5_close_gate.sh
- inc6_tool_gate.sh
- inc7_gate.sh
- kv_crossmode_gate.sh
- launch_defaults_gate.sh
- m2_ultra.csv
- m2_ultra_ts.svg
- m3_max_ts.svg
- m4_max.csv
- m4_max_ts.svg
- manifest_identity_gate.sh
- mem_floor_gate.sh
- memcal_gate.sh
- memdrain_ab_leg.sh
- memdrift_attrib.sh
- memgov_soak.sh
- memresidual_experiment.sh
- mt4_noserial_gate.sh
- mt5_defaults_gate.sh
- mt6_pool_gate.sh
- mtp_ab_stamps.sh
- needle_prompt.py
- needle_sweep.sh
- needle_toolcall_harness.py
- plot_mtp_compare.py
- plot_speed.py
- pro6000_blackwell.csv
- pro6000_blackwell_ts.svg
- pro_model_m3_ultra.csv
- pro_model_m3_ultra_ts.svg
- promessi_sposi.txt
- README.md
- reasoning_effort_gate.sh
- rent_gate.sh
- restore_reject_gate.sh
- serial_lease_gate.sh
- serial_reap_gate.sh
- serial_reclaim_gate.sh
- serial_reserve_gate.sh
- serial_rightsize_gate.sh
- sse_probe_client.py
- sweep_gb10_branchcont_v011q_low.csv
- sweep_gb10_dsparkq_v011q_code_low.csv
- sweep_gb10_dsparkq_v011q_high.csv
- sweep_gb10_dsparkq_v011q_low.csv
- teb_gates.sh
- tool_rerender_gate.sh
- toolcall_depth_gate.sh
- topk_band_gate.sh
- truncate_gate.sh
- update_check_gate.sh
- v010_sweep_overlay.svg
- v011_decode_overlay.svg
- v030_sweep_overlay.svg
- v040_sweep_overlay.svg
- v050_sweep_overlay.svg
- v062_budgets_gate.sh
- vmm_trim_gate.sh
- warm_equal_match_gate.sh
- width384_stamp.sh
- - admit_c_oracle.c
  - agent_c_oracle.c
  - bind_c_oracle.c
  - bind_lookup_c_oracle.c
  - bridge_null_oracle.c
  - bridge_null_stubs.c
  - catalog_c_oracle.c
  - cont_c_oracle.c
  - dist_c_oracle.c
  - dsml_c_oracle.c
  - kv_c_oracle.c
  - kv_c_stubs.c
  - layout_c_oracle.c
  - load_c_oracle.c
  - memgov_c_oracle.c
  - parse_c_oracle.c
  - payload_c_oracle.c
  - render_c_oracle.c
  - render_tools_inc.c
  - retry_c_oracle.c
  - route_c_oracle.c
  - server_c_oracle.c
  - session_c_oracle.c
  - shape_c_oracle.c
  - stream_c_oracle.c
  - tensor_c_oracle.c
  - tokenizer_c_oracle.c
  - tool_stream_c_oracle.c
  - validate_c_oracle.c
  - vocab_c_oracle.c
  - web_c_oracle.c
- - build_essay_prompt.py
  - exact-candidate-top2-plan.json
  - example-argmax-plan.json
- - fetch_official_vectors.py
  - local-golden.vec
  - manifest.json
  - official.vec
  - README.md
- bench_qwen_images.py
- chat_template_live.py
- cuda_graph_fragment_probe.cu
- cuda_graph_layer_array_probe.cu
- cuda_graph_memcpy_probe.cu
- cuda_layer_graph_determinism_probe.sh
- cuda_long_context_smoke.c
- cuda_mtp_proof_matrix.py
- dots3_tokenizer_goldens.inc
- ds4_proof.py
- ds4_proof_plan_hash_smoke.py
- ds4_test.c
- ds4_weight_server_harness_smoke.py
- exaone_tokenizer_cases.txt
- generate_long_context_story_prompt.py
- long_context_essay_prompt.txt
- long_context_security_prompt.txt
- long_context_story_prompt.txt
- mmq_bench_stats.py
- run_mmq_bench.sh
- serial_prefill_regrow_replay_repro.sh
- step37_images_live.py
- step37_logits.py
- test_cuda_artifact_scope.c
- test_cuda_artifact_scope.py
- test_cuda_span_lease.c
- test_dots3_cuda.cu
- test_dots3_loader.c
- test_dots3_resident.c
- test_dots3_tokenizer.c
- test_exaone_batch.c
- test_exaone_forward.c
- test_exaone_kernels.c
- test_exaone_ref.c
- test_exaone_tokenizer.c
- test_glm53_dsa.c
- test_glm53_image.c
- test_glm53_loader.c
- test_glm53_session.c
- test_glm53_vision.c
- test_glm53_vision_loader.c
- test_host_memory_guard.py
- test_inkling_attention.c
- test_inkling_attn_prep.c
- test_inkling_bench.py
- test_inkling_encoders.c
- test_inkling_forward.c
- test_inkling_kernels.c
- test_inkling_linear.c
- test_inkling_loader.c
- test_inkling_media.c
- test_inkling_moe.c
- test_inkling_mtp.c
- test_inkling_mtp_shared.c
- test_inkling_norm.c
- test_inkling_q8_batch.c
- test_inkling_session.c
- test_mmid_fast.cu
- test_model_family_kernels.c
- test_motif3_batch.c
- test_motif3_cuda.cu
- test_motif3_loader.c
- test_motif3_long.c
- test_motif3_reference.c
- test_motif3_resident.c
- test_motif3_tokenizer.c
- test_ple_fp8.py
- test_qwen4exp_batch.c
- test_qwen4exp_draft.c
- test_qwen4exp_gdn.c
- test_qwen4exp_gdn_forward.c
- test_qwen4exp_hc_forward.c
- test_qwen4exp_loader.c
- test_qwen4exp_moe.c
- test_qwen4exp_moe_forward.c
- test_qwen4exp_ple.c
- test_qwen4exp_ple_compute.c
- test_qwen4exp_ple_cuda.cu
- test_qwen4exp_ple_forward.c
- test_qwen4exp_ple_reference.py
- test_qwen4exp_primitives.c
- test_qwen4exp_qsa.c
- test_qwen4exp_qsa_forward.c
- test_qwen4exp_tokenizer.c
- test_qwen4exp_verify.c
- test_qwen_vision_attention.c
- test_qwen_vision_model.c
- test_qwen_vision_norm.c
- test_repack_premapped.cu
- test_solar_fattn.c
- test_solar_forward.c
- test_solar_gates.c
- test_solar_kda.c
- test_solar_kda_chunk.c
- test_solar_kda_prefill.c
- test_solar_kv.c
- test_solar_loader.c
- test_solar_session.c
- test_solar_tokenizer.c
- test_split_gguf.c
- test_step37_checkpoint.c
- test_step37_cont.c
- test_step37_forward.c
- test_step37_loader.c
- test_step37_media.c
- test_step37_mtp.c
- test_step37_norm.c
- test_step37_primitives.cu
- test_step37_session.c
- test_step37_spec.c
- test_step37_state.c
- test_step37_vision.c
- test_step37_vision_ops.cu
- ds4_weight_server.cu
- dspark_trace_replay.py
- host_memory_guard.py
- kernel_scan.py
- plot_solar_open2_20260907.py
- - .gitignore
  - Cargo.toml
  - COMPATIBILITY.md
  - LICENSE-APACHE
  - LICENSE-MIT
  - README.md
  - VENDOR.md
- stb_image.h
.dockerignore
.gitignore
AGENT.md
AGENTS.md
Cargo.lock
Cargo.toml
CHANGELOG.md
CLAUDE.md
CONTRIBUTING.md
download_model.sh
ds4.c
ds4.h
ds4_agent.c
ds4_bench.c
ds4_cli.c
ds4_cuda.cu
ds4_distributed.c
ds4_distributed.h
ds4_eval.c
ds4_glm53_vision_gpu.cuh
ds4_gpu.h
ds4_inkling_gpu.cuh
ds4_iq2_tables_cuda.inc
ds4_kvstore.c
ds4_kvstore.h
ds4_mem_census.h
ds4_mem_gov.h
ds4_metal.m
ds4_model_catalog.h
ds4_ple.c
ds4_ple.h
ds4_server.c
ds4_step37_gpu.cuh
ds4_step37_graph.inc
ds4_step37_vision.inc
ds4_step37_vision_gpu.cuh
ds4_web.c
ds4_web.h
ds4_weight_identity.h
LICENSE
linenoise.c
linenoise.h
Makefile
METAL_DSPARK.md
MODEL_CARD.md
rax.c
rax.h
rax_malloc.h
README.md
rust-toolchain.toml
rustfmt.toml
VERSION

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**DwarfStar** (`ds4`) is a small native inference engine specialized for a
deliberately limited set of large open-weight models. It is self-contained and
deliberately narrow, not a general GGUF runner and not a wrapper around another
runtime. Model loading, prompt rendering, tool calls, model state, KV reuse,
the HTTP server, and the coding agent are built and tested together.

`ds4-dfm-rs` is the independent Rust-host continuation of the DFM line from
[`Baekpica/ds4`](https://github.com/Baekpica/ds4). Its NVIDIA reference machine
is the 128 GB DGX Spark / GB10. It retains the native Metal and CUDA heritage,
and the published RC live evidence is CUDA-based.

As in the original [`antirez/ds4`](https://github.com/antirez/ds4), model
support is intentionally opportunistic. The project follows useful open
weights that fit real personal and workstation-class machines. A family is
supported explicitly, against a known artifact and execution path; it may be
retired when a better model makes it irrelevant.

## So, what can I do with this software?

- Run one of the ten validated model families on a DGX Spark without pulling
  in a general inference framework.
- Serve OpenAI-compatible Chat, Completions, and Responses APIs, Anthropic
  Messages, or use the built-in DeepSeek DSML coding agent.
- Execute official [model Jinja templates](docs/chat-templates.md) for chat
  input, including tool history and REPL turns.
- Use long contexts, persistent KV state, distributed execution, and the
  family-specific acceleration path that was actually validated for the model.
- Serve Qwen3.8 Flash Next Q5 with SSD-PLE sidecars, embedded MTP, and still
  image input.
- Serve K2-Horizon-375B MQ87 with IFM chat/tool syntax on the continuous lane.
- Profile prefill and decode with [ds4-perf](docs/ds4-perf.md),
  calibrate the GPU, compare proved experiments, and retain raw profiler evidence.
- Treat the existing family implementations as rails for a new model or a
  specific machine, while keeping the resulting path small enough to inspect.

## Motivations

- Capable open-weight models now fit on high-end personal machines.
- Routed-expert quantization, compressed or recurrent state, and fast local
  SSDs make very large models and long contexts practical on those machines.
- An inference system specialized for a few models can remain understandable,
  measurable, and aggressively optimized.
- DFM (독자 파운데이션 모델, 독파모) and adjacent families are increasing,
  but their tensor layouts, state machines, prompt protocols, and kernels are
  meaningfully different.
- A Rust host can make those families safer to operate and easier to extend
  without hiding the differences behind a large abstraction layer.

## How to use this project

The original DwarfStar is also a statement about how software can be shipped
in the age of coding agents: a repository can be a working implementation for
the most useful cases and a rail for adapting a new model or hardware setup,
instead of pretending to cover every possible combination.

That remains the intended use here. Start from a validated family, make the
smallest explicit change for the new tensor, state, protocol, or kernel
contract, and rerun the same correctness and performance gates. Coding agents
can make a specialized port much cheaper; they do not replace real artifacts,
hardware measurements, or human ownership of the result.

## AI full disclosure

This line is developed with strong coding-agent assistance, with humans leading
the ideas, scope, testing, and debugging. We say this openly because it shaped
both the Rust migration and the model-family work. It is equally important to
say that DwarfStar would not exist without the largely hand-built work in
[`llama.cpp`](https://github.com/ggml-org/llama.cpp) and GGML, or without the
original DwarfStar and Entrpi CUDA-serving work preserved in this history.

## Why this repository exists

The split is not meant to turn ds4 into a generic framework. It makes a growing
set of DFM and adjacent model families more efficient to extend while keeping
the minimum abstraction that proven families actually share. Rust owns host
lifecycle and serving policy; the native engine stays close to the hardware.

This is therefore **not a rewrite of ds4 in Rust**. The project preserves the
optimized C/CUDA/Metal backend, Git ancestry and authorship, and the full
`antirez → Entrpi → Baekpica` lineage.

## Status

**v0.1.2:** official model Jinja drives Chat, Messages, Responses and CLI
input through one Rust adapter. The [release ledger](docs/releases/v0.1.2.md)
records local artifact and protocol checks, including the
[Inkling Small / MTP checkpoint](docs/inkling-small.md).
DeepSeek V4 retains its encoder exception; see
[template setup and boundaries](docs/chat-templates.md).

The [v0.1.1 ledger](docs/releases/v0.1.1.md) records measured performance
workflows and optional official Qwen FP8 PLE sidecars; the
[FP8 guide](docs/qwen38-ple-fp8.md) covers selection and paired benchmarks.

The first independent Rust-host baseline is **v0.1.0**.
Rust owns the host runtime, policy, serving, KV/state, distributed execution,
observability, and performance orchestration. CUDA/MMQ/VMM remains native.
The [baseline ledger](docs/releases/v0.1.0.md) records production, profiler,
numerical and memory qualification, with exact artifacts and workload limits.

The baseline includes `ds4-perf`, Qwen Session KV restoration, and the
post-RC.4 K2-Horizon MQ87 integration. Supported family/artifact contracts
below remain specific to their validated DGX Spark CUDA paths.

| Item | Baseline evidence |
|---|---|
| Release baseline | `v0.6.5-dfm` (`d02e2a4`) |
| Frozen Qwen C behavior | post-tag cut `4d40d97` |
| Split genesis | annotated tag `ds4-dfm-rs-genesis` (`fe7733f`) |
| Release-tested hardware | NVIDIA DGX Spark / GB10, CUDA |
| Host migration | Rust default binaries; C binaries retained as oracles |
| Native backend | C/CUDA/MMQ/VMM/vision kernels retained |
| License | MIT, inherited notices preserved |

The pre-split campaign finished with 60 logical cells: 57 PASS and three
PASS* cells reproduced on the matching C control, with no Rust-only failure.
The detailed evidence is in
[`SPLIT_READINESS.md`](docs/rust-migration/SPLIT_READINESS.md).

Only explicit, validated GGUF layouts are accepted.

## Design philosophy

- Keep model mechanics visible. Shapes, tensor names, state layouts, prompt
  protocols, and stop rules are family contracts, not plugin metadata.
- Add only the abstraction shared by proven families. Prefer an enum, a table,
  or a narrow function over a runtime plugin system.
- Keep the hot path direct. Family dispatch must not force dynamic dispatch or
  erase kernel-specific information.
- Use Rust where ownership matters: HTTP, admission, scheduling, model/session
  lifetime, KV policy, memory policy, and distributed orchestration.
- Keep CUDA, MMQ, VMM, fused attention, MoE, SSD-PLE, and vision execution
  native. A host-language change is not permission to rewrite kernels.
- Promote behavior only after C/Rust parity, real-model checks, and measured
  performance. Goldens are not refreshed to hide drift.

The practical family-extension contract is intentionally small:

1. identify and validate the GGUF shape;
2. resolve the tensor inventory and bind plan;
3. define tokenizer, prompt, tool, and stop behavior;
4. connect the native state and kernel path through the opaque bridge;
5. define session/KV ownership and serving-lane eligibility;
6. land a focused regression, then loader, forward, API, live, and performance
   evidence.

## Architecture

```text
OpenAI / Anthropic clients, CLI, agent
                    │
                    ▼
┌──────────────────────────────────────────────┐
│ Rust host                                    │
│ HTTP · rendering · admission · scheduler     │
│ model/session lifecycle · KV · memory policy │
│ distributed orchestration                    │
└──────────────────────┬───────────────────────┘
                       │ narrow opaque ABI
                       ▼
┌──────────────────────────────────────────────┐
│ Native backend                               │
│ GGUF mmap · VMM · CUDA Graph · MMQ           │
│ fused attention · MoE · SSD-PLE · vision     │
│ CUDA / Metal / CPU reference                 │
└──────────────────────────────────────────────┘
```

The boundary is [`native/bridge/ds4_bridge.h`](native/bridge/ds4_bridge.h),
not a generated binding of the engine internals. Safe Rust never receives
CUDA streams, device pointers, graph handles, VMM allocation handles, or raw
native structs.

| Area | Owner |
|---|---|
| API parsing, rendering, streaming, routing | Rust (`ds4-server`) |
| GGUF identification, validation, inventory, bind plan | Rust (`ds4-core`) |
| Model/session handles and lifetime policy | safe Rust over opaque native handles |
| KVC metadata, persistence policy, cross-host codecs | Rust (`ds4-kv`) |
| Distributed protocol and orchestration | Rust (`ds4-dist`) |
| Profiling orchestration, normalization, diagnosis | Rust (`ds4-perf`, separate process) |
| Prefill/decode NVTX annotations | Rust (`ds4-cli`, optional official NVIDIA SDK) |
| CUDA/VMM/MMQ/graphs/attention/MoE/vision | native C/CUDA |
| Metal and CPU reference paths | inherited native backend |
| C parity executables and `ds4-eval` | retained release oracles |

The host uses blocking sockets, threads, channels, mutexes, and condition
variables. There is no async framework added merely because the host is Rust.
See [`ARCHITECTURE.md`](docs/rust-migration/ARCHITECTURE.md) and
[`FFI_CONTRACT.md`](docs/rust-migration/FFI_CONTRACT.md) for the full boundary.

## Supported hardware and backends

| Backend | Documented scope |
|---|---|
| NVIDIA DGX Spark / GB10 | Release target. The split-era full matrix, long-context, Qwen image/MTP, ABBA, and soak gates ran here; RC.3's agent-serving matrix, RC.4's GLM Q2 text/vision gates, and the K2-Horizon-375B MQ87 32K CLI/HTTP gates also ran here. |
| Other NVIDIA CUDA systems | Source path retained through `make cuda-generic` or an explicit `CUDA_ARCH`; not covered by the RC's full live matrix. |
| macOS Metal | Inherited source/build path retained; not part of the DFM RC live gate. |
| CPU | Reference and diagnostics only, not a production performance backend. |

The recorded RC host used CUDA 13.3.73, driver 610.43.02, Linux
6.17.0-1031-nvidia, and Rust 1.98.0. Do not generalize its measurements to a
different GPU or quant without rerunning the same gate.

## Supported model families

Every family below has an explicit architecture selector, validator, binder,
tokenizer/chat contract, state lifecycle, and native execution path.

| Family | GGUF architecture | Documented scope |
|---|---|---|
| DeepSeek V4 Flash / PRO | `deepseek4` | Flash is the main live oracle; DeepSeek MTP and DSpark sidecars. |
| Solar Open2 250B | `solar-open2` | Recurrent KDA state, compressed GQA KV, persistent banks. |
| K-EXAONE 236B A23B | `exaone-moe` | LLLG full/sliding GQA KV and persistent banks. |
| Motif-3 | `motif3` | Latent KV, rotated `k_pe`, SWA rings, persistent banks. |
| dots3-note Preview | `dots3note` (`dots3-note` accepted) | Dual-geometry latent state; current live serving path is serial. Tensor-core prefill attention / MLA GEMMs and split-K decode attention since 2026-09-06. |
| Qwen3.8 Flash Next SSD-PLE | `qwen4exp` | Q5 main GGUF + BF16 or [official FP8 SSD-PLE](docs/qwen38-ple-fp8.md), embedded MTP, N-bank Rust scheduling, still-image input; one- and two-bank live gates. |
| GLM 5.3 Flash | `glm5-next` | Q2 single-file GGUF plus the explicit vision sidecar; CUDA serial serving on one DGX Spark. |
| K2-Horizon 375B A23B | `k2-horizon` | Four-shard MQ87 GGUF; IFM BPE/XML tools; continuous 32K one-bank serving on one DGX Spark. |
| Inkling Small | `inkling` | MQ85GB + optional eight-layer MTP-BF16; serial CUDA text/image/audio input and text output. [HTTP checks and limits](docs/inkling-small.md), [GB10 performance](docs/inkling-optimization-2026-09-11.md). |
| Step 3.7 Flash | `step35` | Nine-shard MQ83, optional three-block Q8 MTP and F16 vision; CUDA text/image serving, opt-in text banks with MTP, partial fork and disk KV. [Serving limits](docs/step37-serving-2026-09-13.md), [capped-clock A/B](docs/step37-optimization-2026-09-13-r3.md). |

The current family contract and measured model-specific limits are documented
in [`ds4-dfm-model-families.md`](docs/ds4-dfm-model-families.md). Arbitrary
GGUFs, alternate tensor layouts, and unlisted architectures are rejected.

### Model Zoo

See also the
[`DS4-Mixed-Quant-for-Spark`](https://huggingface.co/collections/Baekpica/ds4-mixed-quant-for-spark)
collection. Support remains limited to the validated layouts described above.

| Model | GGUF artifact | Artifact by |
|---|---|---|
| DeepSeek V4 Flash / PRO | [`antirez/deepseek-v4-gguf`](https://huggingface.co/antirez/deepseek-v4-gguf/tree/main) | [`antirez`](https://huggingface.co/antirez) |
| Solar Open2 250B | [`Baekpica/Solar-Open2-250B-Mixed-Quant-GGUF`](https://huggingface.co/Baekpica/Solar-Open2-250B-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| K-EXAONE 236B A23B | [`Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF`](https://huggingface.co/Baekpica/K-EXAONE-236B-A23B-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| Motif-3 | [`Baekpica/Motif-3-Mixed-Quant-GGUF`](https://huggingface.co/Baekpica/Motif-3-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| dots3-note Preview | [`Baekpica/dots3-note-prev-Mixed-Quant-GGUF`](https://huggingface.co/Baekpica/dots3-note-prev-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| Qwen3.8 Flash Next SSD-PLE | [`Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF`](https://huggingface.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| GLM 5.3 Flash | [`GLM-5.3-Flash-Q2.gguf`](https://huggingface.co/antirez/glm-5.3-flash-gguf/blob/main/GLM-5.3-Flash-Q2.gguf) + [`vision encoder`](https://huggingface.co/antirez/glm-5.3-flash-gguf/blob/main/GLM-5.3-Flash-Vision-Encoder.gguf) | [`antirez`](https://huggingface.co/antirez) |
| K2-Horizon 375B A23B | [`Baekpica/K2-Horizon-375B-A23B-Mixed-Quant-GGUF`](https://huggingface.co/Baekpica/K2-Horizon-375B-A23B-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |
| Inkling Small | [`MQ85GB`](https://huggingface.co/Baekpica/Inkling-Small-Mixed-Quant-GGUF/tree/main/MQ85GB) + optional [`MTP-BF16`](https://huggingface.co/Baekpica/Inkling-Small-GGUF/tree/main/MTP-BF16) | [`Baekpica`](https://huggingface.co/Baekpica) |
| Step 3.7 Flash | [`MQ83 + MTP + vision`](https://huggingface.co/Baekpica/Step-3.7-Flash-Mixed-Quant-GGUF) | [`Baekpica`](https://huggingface.co/Baekpica) |

### Qwen release scope

The initial Rust RC qualification covered:

- [`MQ-Q5-SSD-PLE-BF16`](https://huggingface.co/Baekpica/Qwen3.8-Flash-Next-Mixed-Quant-SSD-PLE-GGUF), three main GGUF shards;
- four shared BF16 SSD-PLE sidecars referenced by that Q5 layout;
- embedded MTP with `--mtp-draft 2`;
- text and base64 PNG/JPEG input on the three message APIs;
- 196,608 two-bank serving and 262,144 one-bank serving, in addition to the
  earlier exact/configured 262,144-token gates.

Q6, original safetensors, and a resident BF16 GGUF were not release gates and
are not implied by this claim.

For the optional FP8 PLE sidecar with the existing base and
[Uncensored](https://huggingface.co/Baekpica/Qwen3.8-Flash-Next-Uncensored-Mixed-Quant-SSD-PLE-GGUF)
main GGUFs, see [selection, validation and paired 64K sweeps](docs/qwen38-ple-fp8.md).

Rust normalizes ordered image parts, bounds and owns payload bytes, places
image tokens, and owns decoded-pixel cache identity. Decoding reuses the pinned
[`vendor/stb_image.h`](vendor/stb_image.h) through a narrow native image ABI;
vision and CUDA execution stay native. No general multimedia layer or Rust
image dependency was added.

Image limits match the frozen C behavior:

- user messages only;
- PNG or JPEG data URIs only;
- at most four images;
- at most 10 MiB decoded per image and 20 MiB per request;
- remote URLs, files, SVG, GIF, WebP, malformed base64, and invalid image
  content are rejected.

See [`QWEN_V065_RESTAMP_2026-08-31.md`](docs/rust-migration/QWEN_V065_RESTAMP_2026-08-31.md)
and [`qwen38-image-input-spec.md`](docs/qwen38-image-input-spec.md).
Measured image latency and agent checks:
[`qwen38-image-2026-09-07.md`](docs/qwen38-image-2026-09-07.md).

### GLM 5.3 Flash release scope

RC.4 follows the explicit GLM 5.3 Flash graph and vision implementation in
the official [`antirez/ds4`](https://github.com/antirez/ds4) upstream, pinned
for this port at
[`110afdd`](https://github.com/antirez/ds4/commit/110afdd8886586f18fc9b28bc5533152dd10e728).
The Rust host keeps the KDA, DSA, hyper-connection mixing, MoE, and
[`vision encoder`](https://github.com/antirez/ds4/blob/110afdd8886586f18fc9b28bc5533152dd10e728/ds4_glm53_vision_gpu.cuh)
execution native.

The verified artifact set is exactly:

- `GLM-5.3-Flash-Q2.gguf` — 96,505,816,384 bytes;
- `GLM-5.3-Flash-Vision-Encoder.gguf` — 1,127,280,960 bytes, SHA-256
  `ae23e14c6979e889051b2e4a39351abcdafb161e18e606fae4d8c40095a4bf3a`.

The following command reproduces the RC.4 live smoke shape:

```sh
MODEL_DIR=/path/to/GLM-5.3-Flash-Mixed-Quant-GGUF

./ds4-server --cuda \
  -m "$MODEL_DIR/GLM-5.3-Flash-Q2.gguf" \
  --vision "$MODEL_DIR/GLM-5.3-Flash-Vision-Encoder.gguf" \
  --model-id GLM-5.3-Flash-Q2 \
  -c 256 -n 8 \
  --host 127.0.0.1 --port 8000
```

The current GLM graph is serial and has an explicit 2,048-token context cap.
OpenAI Chat text and inline PNG image requests were served live on one DGX
Spark; model-free parsing gates also cover the equivalent Responses and
Anthropic inline-image forms. PNG and JPEG are accepted, with at most four
images per request. Q4, FP8, full GLM 5.3, Metal, ROCm, distributed serving,
SSD streaming, continuous batching, and speculative MTP were not RC.4 gates
and are not implied by this support entry.

### K2-Horizon-375B release scope

This branch follows the IFM
[`K2-Horizon-375B-A23B`](https://huggingface.co/IFM/K2-Horizon-375B-A23B)
graph: 61 full-attention GQA layers, partial NeoX RoPE on 64 of 128 dims,
three leading dense MLPs, sigmoid top-8 routing with one shared expert, and
no MTP. Execution stays native. The GGUF architecture is `k2-horizon`; it
does not widen the K-EXAONE LLLG/QK-norm contract.

The verified artifact set is exactly the public MQ87 split, 93,091,935,552
bytes (86.698621 GiB) across four shards:

- `K2-Horizon-375B-A23B-MQ87-00001-of-00004.gguf`
- `K2-Horizon-375B-A23B-MQ87-00002-of-00004.gguf`
- `K2-Horizon-375B-A23B-MQ87-00003-of-00004.gguf`
- `K2-Horizon-375B-A23B-MQ87-00004-of-00004.gguf`

Expected inventory: 842 tensors (`Q8_0=429`, `F32=239`, `IQ1_S=100`,
`IQ2_XXS=50`, `IQ1_M=16`, `IQ2_XS=8`). Official FP8 checkpoints are not a
runtime input.

On GB10, whole-map `cudaHostRegister` of the 86.70 GiB mmap fails. The
existing VMM materializer then promotes every unit (95/95, 0 cold) so CUDA
graphs never capture the unregistered mmap. Default `DS4_MEMGOV=enforce`
accepted the 32K first boot on this host.

The following command reproduces the live serving shape with in-process VMM.
External weight-owner import is not qualified for this artifact; see the
[release ledger](docs/releases/v0.1.0.md).

```sh
MODEL=/path/to/K2-Horizon-375B-A23B-Mixed-Quant-GGUF/K2-Horizon-375B-A23B-MQ87-00001-of-00004.gguf

./ds4-server --cuda \
  -m "$MODEL" \
  --model-id K2-Horizon-375B-A23B-MQ87 \
  -c 32768 --cont-width 1 \
  --host 127.0.0.1 --port 8000
```

CLI 32K raw-token smoke returned token `33785` with default memgov. HTTP
Chat, XML tool call/result continuation, streaming, and concurrent requests
passed on the same one-bank 32K setup. Official IFM `high` thinking is the
gated path. The 524,288-token metadata context, `low`/`medium` think
variants, other quants, Metal, ROCm, and distributed serving were not
gates and are not implied by this support entry.

## Build

The repository pins Rust 1.98.0 with `rustfmt` and `clippy` in
[`rust-toolchain.toml`](rust-toolchain.toml). CUDA builds also require a local
CUDA toolkit and C/C++ build tools.

```sh
git clone https://github.com/Baekpica/ds4-dfm-rs.git
cd ds4-dfm-rs
make cuda-spark
```

Important build targets:

| Command | Result |
|---|---|
| `make cuda-spark` | DGX Spark / GB10 CUDA build with the `sm_121a` code path |
| `make cuda-generic` | CUDA build for the detected local GPU |
| `make cuda CUDA_ARCH=sm_N` | CUDA build with an explicit architecture |
| `make` on macOS | Metal build |
| `make cpu` | CPU reference/diagnostic build |

The production names remain `ds4`, `ds4-server`, `ds4-bench`, and
`ds4-agent` for this parity RC. They are Rust-host binaries. Their
`ds4-c`, `ds4-server-c`, `ds4-bench-c`, and `ds4-agent-c` counterparts are C
oracles. `ds4-eval` is still the C extractor oracle. The old `*-rs` names are
deprecated build aliases, not a second runtime.

`./ds4-server --version` reports the independent repository version from the
Rust package; `make print-version` reports the Git-derived native build stamp.

## Quick start

Models are not bundled. Pass the first shard of a supported split GGUF with
`-m`.

```sh
MODEL=/path/to/supported-model-00001-of-000NN.gguf

./ds4-server \
  --cuda \
  -m "$MODEL" \
  -c 131072 \
  --host 127.0.0.1 \
  --port 8000 \
  --model-id local-model \
  --no-update-check
```

Then verify discovery, state, and a real generation:

```sh
curl -s http://127.0.0.1:8000/v1/models
curl -s http://127.0.0.1:8000/v1/stats
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local-model","messages":[{"role":"user","content":"Hello"}]}'
```

`./ds4 --help`, `./ds4-server --help`, `./ds4-bench --help`, and
`./ds4-agent --help` are the authoritative flag references.

### Weight owner and worker

On unified-memory systems, a weight owner keeps one VMM allocation alive while
inference workers restart. Keep the manifest path short because its Unix
socket is `<manifest>.sock`.

```sh
MODEL=/path/to/supported-model.gguf
MANIFEST=/tmp/ds4-weights.manifest

./ds4_weight_server \
  --base "$MODEL" \
  --manifest "$MANIFEST" \
  --backend vmm \
  --scope base \
  --reserve-gb 32
```

Wait for both `broker listening` and `ready manifest=...`, then start the
worker in another durable session:

```sh
DS4_CUDA_WEIGHT_IPC_MANIFEST="$MANIFEST" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
./ds4-server --cuda -m "$MODEL" -c 196608 \
  --host 127.0.0.1 --port 8000 --no-update-check
```

Qwen Q5 release runs additionally set a bounded SSD-PLE cache. Size it from
the prefill chunk: a chunk's sixteen 320-byte PLE rows per token land on
about 1.08 4 KiB pages each, so an 8,192-token chunk needs ~553 MiB of pages,
and the engine prefetches the *next* chunk's pages while the current chunk's
decoder layers run (`DS4_QWEN_PLE_NO_LOOKAHEAD=1` disables that). One prefill
stream therefore wants at least one chunk in cache (1024 MiB with slack); two
banks that alternate chunks want two (2048 MiB, the maximum). Sixteen page
workers already saturate the sidecar reads at ~90K IOPS in bursts that overlap
compute, so more workers do not help. A prompt's first chunk has nothing
queued for it, so every prompt opens with a 2,048-row chunk whose remaining
decoder layers hide the reads of the full-size chunk behind it; prompts
shorter than two opening chunks stay one chunk, since a short trailing
chunk costs more than the reads it hides
(`DS4_QWEN_PREFILL_OPENING` sets the opening rows; `0` opens at the chunk
cap). This reference shape asks the shared Rust scheduler for two persistent
banks:

```sh
DS4_QWEN_BATCH=1 \
DS4_QWEN_PLE_CACHE_MB=2048 \
DS4_QWEN_PLE_WORKERS=16 \
DS4_QWEN_PREFILL_CHUNK=8192 \
DS4_SERVER_COALESCE_MAX=2 \
DS4_CUDA_WEIGHT_IPC_MANIFEST="$MANIFEST" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
./ds4-server --cuda -m "$MODEL" -c 196608 --mtp-draft 2 \
  --cont-width 2 --host 127.0.0.1 --port 8000 --no-update-check
```

### Qwen YaRN long contexts

Qwen contexts through 262,144 tokens retain the native factor-1 rotary path.
Larger server contexts select a static YaRN factor from the requested context:
factor 2 through 524,288, factor 3 through 786,432, and factor 4 through
1,048,576. This follows the
[`Qwen3.8-Flash-Next` 1M recipe](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8#processing-ultra-long-texts)
and the
[`transformers` YaRN equations](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_rope_utils.py);
the underlying method is described in the
[`YaRN` paper](https://arxiv.org/abs/2309.00071).

The 1M configuration uses one bank and a smaller prefill chunk:

```sh
DS4_SESSION_GRAPH_FIT=0 \
DS4_QWEN_BATCH=1 \
DS4_QWEN_PLE_CACHE_MB=512 \
DS4_QWEN_PLE_WORKERS=16 \
DS4_QWEN_PREFILL_CHUNK=256 \
DS4_SERVER_COALESCE_MAX=1 \
DS4_SERVER_FORK=0 \
DS4_SERVER_FORK_PARTIAL=0 \
DS4_CUDA_WEIGHT_IPC_MANIFEST="$MANIFEST" \
DS4_CUDA_WEIGHT_IPC_SCOPE=base \
./ds4-server --cuda -m "$MODEL" -c 1000000 -n 256 --cont-width 1 \
  --host 127.0.0.1 --port 8000 --no-update-check
```

`DS4_SESSION_GRAPH_FIT=0` is an explicit fit-check override, not a claim that
the requested context fits the machine. On a 128 GB DGX Spark, the Q5+Sidecar
run recorded the following staged boundary on 2026-09-01:

| Configured context | YaRN factor | Largest prompt run | Result |
|---:|---:|---:|---|
| 196,608 | 1 | text and JPEG smoke | PASS, native-context regression |
| 524,288 | 2 | 524,240 tokens | HTTP 200, 215.4 prefill tok/s, zero census faults |
| 1,000,000 | 4 | 300,040 tokens | HTTP 200, 261.4 prefill tok/s, text/JPEG smoke, zero census faults |

The 524K run peaked at about 30.6 GiB in the worker and finished 47 tokens
below its context cap. A complete 1M-token prompt is **not** claimed: its
53.56 GiB graph plan plus the roughly 80.65 GiB weight owner exceeds the
machine's 121.63 GiB usable unified-memory budget. Use the native context for
ordinary short requests because static YaRN can reduce short-context quality.

Large GGUFs can exhaust unified or system memory. During validation, load one
production model at a time, observe accelerator activity and per-process memory
with tools available on your platform, and confirm serving processes have
exited before reclaiming host resources.

## HTTP compatibility

| Surface | Endpoint |
|---|---|
| OpenAI Chat Completions | `POST /v1/chat/completions` |
| OpenAI Completions | `POST /v1/completions` |
| OpenAI Responses | `POST /v1/responses` |
| Anthropic Messages | `POST /v1/messages` |
| Model discovery | `GET /v1/models` |
| Runtime state | `GET /v1/stats` and `GET /metrics` |

Buffered and SSE streaming forms preserve their surface-native response
objects, tool calls, reasoning fields, finish semantics, and error envelopes.
The server has serial, continuous, and static lanes; set
`DS4_SERVER_CONTINUOUS=0` to force the static/serial route used by the C
compatibility gate.

The continuous lane is width-generic: the Rust host schedules up to the
configured and native-fitted bank count, serializes work when only one bank is
available, and refills free banks from the live queue without waiting for the
longest row. Admission limits, disconnect cancellation, stream heartbeats and
typed failures, shutdown propagation, and cumulative usage accounting live in
the shared Rust serving path. Each model family still supplies its explicit
state and KV contract; a configured width is not a claim that every model and
context fits that width on a given machine.

`DS4_SERVER_MAX_CLIENTS` (default 256) reserves client capacity before request
bodies are read, so slow or oversized ingress cannot consume an unbounded
number of reader threads.

Resident-bank protection and SSD checkpoint eligibility are independent:
`DS4_SERVER_PIN_MIN_TOKENS` defaults to 65,536, while
`DS4_SERVER_PERSIST_MIN_TOKENS` defaults to 8,192. Lowering the persistence
threshold does not pin shallow sessions in memory.

Qwen and GLM image content is accepted in the existing API-native shapes:

```text
Chat:      {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}
Responses: {"type":"input_image","image_url":"data:image/jpeg;base64,..."}
Anthropic: {"type":"image","source":{"type":"base64","media_type":"image/jpeg","data":"..."}}
```

The exact route table, supported fields, and explicit refusals are in
[`ds4-api-surface-matrix.md`](docs/ds4-api-surface-matrix.md).

### Compatibility boundary

The project attempts to preserve these public contracts through `v0.x`:

- core `ds4-*` CLI flags and `DS4_*` environment variables;
- HTTP request/response behavior and observable sampling semantics;
- KVC files, EXT_TOOL_MAP, KTM trailers, recurrent payloads, and image replay
  keys across C-save/Rust-load and Rust-save/C-load;
- explicit distributed byte codecs.

The following are internal and may change during `v0.x`: crate APIs, Rust
types, file layout, scheduler internals, and the native bridge ABI. Native and
Rust objects must be built from the same commit. Distributed transport has no
wire version, encryption, or authentication; use matching binaries on a
trusted network.

The HTTP server is one trust domain and does not provide tenant isolation or
authentication. Put it behind an authenticating proxy or run one server per
trust domain when clients are not mutually trusted.

## Performance and release evidence

The [September 12 Solar campaign](docs/solar-open2-optimization-2026-09-12.md)
closed after four candidates without retaining a new kernel. It records
guarded 8K/64K baseline measurements, numerical and timing acceptance
results, and the limits of the memory guard.

> **Solar Open2 250B MXQ-v1 · one DGX Spark:** verified cold prefill reaches **1,108.09 tok/s at 8K (+1.39%)** and **780.02 tok/s at 64K (+1.02%)** after the fused MoE pass. Interleaved medians of three; all 196,608 frontier logits and 64 greedy IDs match byte for byte. Decode differs by less than 0.1%. [Protocol, raw data and campaign limits](docs/solar-open2-optimization-2026-09-07.md).

![Solar Open2 verified cold comparison and baseline-only sweep through 64K](https://raw.githubusercontent.com/Baekpica/ds4-dfm-rs/main/docs/solar-open2-2026-09-07-throughput.png)

*The lower curves show the baseline only. The campaign ended after repeated
host freezes; the second prefill candidate and decode drafts are excluded.
No optimized sweep or new agent-serving performance result is claimed.*

![Qwen3.8 Flash Next Q5 paired BF16 and FP8 PLE throughput](https://raw.githubusercontent.com/Baekpica/ds4-dfm-rs/main/docs/qwen38-ple-fp8-base.png)

*Base Q5 on one DGX Spark / GB10, with only the PLE sidecar changed:
**1,245.1 → 1,323.1 tok/s prefill (+6.3%)** and
**28.59 → 28.93 tok/s decode (+1.2%)**. Median of three run means per format;
curves show per-frontier medians and observed min/max bands. The original
card protocol uses one warm session, 2,048-token incremental prefills through
64K and 128 greedy tokens per frontier, aligned-Q8 owner, 2 GiB PLE cache and
16 workers. MTP draft 2 was requested; one BF16 run autoquenched and remains
in the data. [Base/Uncensored results, exact protocol and limits](docs/qwen38-ple-fp8.md).*

The 2026-09-06 prefill rounds
([`docs/qwen38-prefill-2026-09-06.md`](docs/qwen38-prefill-2026-09-06.md))
moved the cold single-shot `ds4-bench` prefill from 1,214.8 to **1,362.1 tok/s**
at 8,192 tokens and 1,382.7 to **1,439.7 tok/s** at 65,536 tokens
(`main` `0510117` -> `974d706`: opening chunk, MoE glue, one-pass block
output).  Three further rounds on `feature/qwen-prefill-opt-20260906-r4`
(`d9989bb` / `c00eacd` / `abdf25c`: D2R for K=2560 qkv/z/q, one HC-mix Q8
emit, o_proj K=6144 onto D2R) took the same-hour cold medians on the
aligned-Q8 owner to **1,431.5 tok/s** at 8K and **1,554.8 tok/s** at 64K.

The 2026-09-07 rounds
([`docs/qwen38-prefill-2026-09-07.md`](docs/qwen38-prefill-2026-09-07.md):
SwiGLU quantized straight into the fused expert-down, four barriers per
fused-QSA tile with conflict-free partial stores, SSD-PLE gather leased in
16,384-row tiles under one lock; `ef37468` -> `6e036c4`) take the same
protocol from 1,429.9 to **1,504.6 tok/s** at 8K (+5.2 %), from 1,557.2 to
**1,648.4 tok/s** at 64K (+5.9 %) and from 1,439.9 to **1,513.0 tok/s** at
196,608 tokens (+5.1 %, one run each), base and final interleaved in the same
hour; on the production two-bank server shape the 8,259-token repeated prompt
went 1,481.7 -> **1,559.9 tok/s** (+5.3 %) and the 7,937-token cold-PLE
markdown prompt 1,555.9 -> **1,652.6 tok/s** (+6.2 %), with the 175-token
greedy continuation byte-identical before and after.  Every adopted round is
bit-identical to the kernels it replaces on the fixtures, and the 8,192-token
frontier logits of the final binary match the base binary's byte for byte.

The original split gate claims parity class, not a universal speedup.

| Gate | Recorded result on DGX Spark / GB10 |
|---|---|
| Full host/family matrix | 57 PASS + 3 C-reproduced PASS*, 0 FAIL, 0 BLOCKED |
| Qwen exact 262K | 248,320 finite logits, same argmax, zero packed-f32 mismatches; Rust prefill 99.64% of C |
| Qwen text ABBA | Rust/C mean: 99.97% prefill, 100.00% decode, +0.95% TTFT, +4.40% host HWM |
| Qwen image ABBA | Rust/C mean: 100.20% prefill, +0.32% TTFT |
| Qwen soak | 7,202.3 s, 3,610/3,610 requests, 158 width-2 barriers, 79 image requests, zero request/census/governor failures |
| GLM 5.3 Q2 + vision smoke | Exact Q2 and vision sidecar: native 16-image-token prefill with finite logits; Rust text and PNG Chat requests returned HTTP 200 at context 256. |
| K2-Horizon-375B MQ87 | Four-shard 86.70 GiB MQ87: CLI 32K raw-token `33785`; default memgov 95/95 VMM promote; HTTP Chat/tool/stream/concurrent on one 32K bank. |
| dots3-note MQ87 (2026-09-06 rounds) | 8,192-token cold prefill 278.3 → 604.3 tok/s (+117 %), greedy decode 11.66 → 16.78 tok/s (+44 %) on one DGX Spark, serial lane, same-binary kill-switch A/B; frontier logits same argmax / top-10 10/10, 64 greedy IDs identical; resident CPU-reference gate passed (`docs/dots3-optimization-2026-09-06.md`). |
| Step 3.7 Flash MQ83 (300–2200 MHz cap) | Fresh matched 3-pair A/B: 2048+64 Prefill **1194.64 tok/s** (+0.72%), MTP draft-3 Decode **22.77 tok/s** (+4.02%), exact logits/tokens. Separate 16384+64: Prefill **1290.01** (+7.22%), MTP Decode **16.34** (+14.59%, changed trajectory/acceptance). Default chunk 4096; smaller overrides for bank/image memory. Two Prefill and two Decode rounds, raw samples and limits: [capped campaign](docs/step37-optimization-2026-09-13-r3.md). Earlier uncapped results are historical. |

The Qwen measurements used only the Q5+Sidecar artifact, fresh sequential C
and Rust processes, and the conditions recorded in the evidence documents.
The three PASS* cells are engine gaps E-2, E-3, and E-6 reproduced on C; they
are not hidden Rust failures. See
[`PARITY_MATRIX.md`](docs/rust-migration/PARITY_MATRIX.md),
[`ENGINE_GAPS.md`](docs/rust-migration/ENGINE_GAPS.md), and
[`SPLIT_READINESS.md`](docs/rust-migration/SPLIT_READINESS.md).

RC.3 then revalidated the changed agent-serving paths on that same Qwen
Q5+Sidecar artifact. At 262,144 context with one bank, a replayed 140-token
turn reported 116 cached and 24 computed tokens, and six barrier-released
requests completed successfully in FIFO order. At 196,608 context with two
banks, rolling refill let a later short request finish in 4.90 seconds while
the earlier long request finished in 14.53 seconds; the final live counters
reported 19 completed, zero failed, zero canceled, and zero continuous or
memory-census faults. OpenAI Responses and Anthropic streaming tool-output
continuations both resumed their owning bank with cache hits. The two-bank
run retained about 13.0 GiB available host memory.

This was a targeted RC.3 regression matrix, not a second long soak. The
7,202.3-second Qwen-only soak in the table predates these host changes and was
not rerun for RC.3.

The GLM row is an RC.4 correctness and serving smoke, not a throughput or
long-context claim. The K2 row is a Spark correctness and 32K serving smoke
on this branch, not a tagged RC or a 524K live claim.

## Testing

Host checks are model-free after their C parity oracles are built:

```sh
cargo fmt --all -- --check
cargo clippy --workspace --all-targets --locked

make -j1 test-kv-parity
make -j1 test-web-parity
make -j1 test-dist-parity
make -j1 test-server-parity
make -j1 test-catalog-parity
make -j1 test-tokenizer-parity
make -j1 test-session-parity
make -j1 test-agent-parity

cargo test --workspace --locked -- --test-threads=1
cargo check --workspace --all-targets --locked
```

The order matters on a clean checkout: the parity targets create the C oracle
executables consumed by the workspace tests. GitHub Actions runs this host-only
set. It does not pretend that a hosted CPU runner proves CUDA behavior.

Shared CUDA checks include:

```sh
make -j1 test-model-family-kernels
make -j1 test-mmq-parity
./ds4-eval --self-test-extractors
```

Family loaders, real-model forwards, long-context runs, OPP-C, ABBA, and soak
gates need the matching models and release hardware. The
[migration evidence](docs/rust-migration/README.md) preserves the original
protocols; the [v0.1.0 ledger](docs/releases/v0.1.0.md) records baseline
qualification and the [v0.1.1 ledger](docs/releases/v0.1.1.md) records the
performance workflow and FP8 additions. The [v0.1.2 ledger](docs/releases/v0.1.2.md)
records shared Jinja and Inkling scope. See [CONTRIBUTING.md](CONTRIBUTING.md)
for the validation workflow.

## Repository layout

| Path | Purpose |
|---|---|
| `crates/ds4-core` | safe model/session host, GGUF catalog, tokenizer, bind and validation |
| `crates/ds4-server` | HTTP surfaces, routing, scheduling, streaming, tools |
| `crates/ds4-kv` | KVC format and persistence policy |
| `crates/ds4-dist` | distributed codecs and runtime |
| `crates/ds4-cli` | CLI, bench, and agent hosts |
| `crates/ds4-web` | blocking agent web helpers |
| `crates/ds4-perf` | standalone profiling orchestration and diagnosis |
| `crates/ds4-sys` | narrow unsafe FFI and OS adapters |
| `native/bridge` | opaque Rust/native boundary |
| `ds4.c`, `ds4_cuda.cu`, `cuda/`, `metal/` | native engine and kernels |
| `tests/parity` | C behavior oracles consumed by Rust tests |
| `docs/README.md` | current guides, dated evidence and design index |
| `docs/releases` | release definitions and qualification gates |
| `docs/rust-migration` | current boundary contracts and frozen migration evidence |

## Lineage

The repository is independent on GitHub, but its code history is continuous:

```text
antirez/ds4
    ↓
Entrpi/ds4
    ↓
Baekpica/ds4 (DFM edition)
    ↓
Baekpica/ds4-dfm-rs
```

The split preserved Git ancestry, authors, the MIT license, vendor provenance,
and the `v0.6.5-dfm` baseline tag. It did not filter, squash, or relabel the
project as a clean-room implementation. The replaced target scaffold remains
recoverable at `pre-genesis-scaffold-b01d1fa`.

See [`docs/LINEAGE.md`](docs/LINEAGE.md) for the exact refs and ongoing
upstream-port policy.

## Contributing

Read [`CONTRIBUTING.md`](CONTRIBUTING.md) before sending a change.

- Keep changes focused and leave a regression that fails without them.
- A model-family addition must prove loader, tokenizer, tensor binding,
  forward behavior, API rendering, state/KV lifecycle, and its real native
  path.
- CUDA changes need correctness and speed evidence on the affected path.
- Do not introduce a generic plugin layer, a second CUDA stack, C++ host code,
  or a large async runtime without a measured problem and a separate decision.
- Record model revision, quant, commit, hardware, CUDA, context, width, KV mode,
  and thermal conditions with performance results.

## Documentation

- [Documentation index](docs/README.md) — current guides, dated evidence and design records
- [v0.1.0 release ledger](docs/releases/v0.1.0.md) — qualification, evidence and scope
- [v0.1.1 release ledger](docs/releases/v0.1.1.md) — performance workflow and FP8 PLE
- [v0.1.2 release ledger](docs/releases/v0.1.2.md) — official Jinja and Inkling checkpoint
- [`CHANGELOG.md`](CHANGELOG.md) — inherited and fork-side release history
- [`docs/LINEAGE.md`](docs/LINEAGE.md) — repository provenance and split refs
- [`docs/rust-migration/SPLIT_READINESS.md`](docs/rust-migration/SPLIT_READINESS.md) — genesis decision and immutable evidence
- [`docs/rust-migration/ARCHITECTURE.md`](docs/rust-migration/ARCHITECTURE.md) — host/native ownership
- [`docs/rust-migration/FFI_CONTRACT.md`](docs/rust-migration/FFI_CONTRACT.md) — opaque ABI rules
- [`docs/ds4-dfm-model-families.md`](docs/ds4-dfm-model-families.md) — model-family runtime details
- [`docs/ds4-api-surface-matrix.md`](docs/ds4-api-surface-matrix.md) — API and serving-lane contract
- [`cuda/mmq/VENDOR.md`](cuda/mmq/VENDOR.md) — llama.cpp/GGML kernel provenance
- [`misc/proof-harness/README.md`](misc/proof-harness/README.md) — native proof harness

## License and acknowledgements

`ds4-dfm-rs` remains MIT licensed; see [`LICENSE`](LICENSE). Existing
copyright notices are preserved.

The project stands on the original work in
[`antirez/ds4`](https://github.com/antirez/ds4), the CUDA and batched-serving
work in [`Entrpi/ds4`](https://github.com/Entrpi/ds4), and the DFM family and
Rust-host work developed in [`Baekpica/ds4`](https://github.com/Baekpica/ds4).

It also depends on the GGUF ecosystem, quantization formats, engineering
knowledge, and selected MIT-licensed kernel code from
[`llama.cpp`](https://github.com/ggml-org/llama.cpp) and GGML. Their notices
and the exact vendored kernel pin remain in this tree.

ds4-dfm-rs

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README