Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4_tools_eval

KO | EN

gitlite — search

#web #svg #ai #animation #devops #swift #bash #agent #artificial-intelligence #editor

Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4_tools_eval

(★ 9)

tool-eval-bench: Qwen3.6-27B GGUF Q8_K_XL vs NVIDIA NVFP4 — head-to-head tool-calling quality comparison

File Explorer

ZIP Download

README.md
Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4.html

# Use via CDN

jsDelivr

jsDelivr serves any public GitHub repository as a CDN with zero setup. Pick a version and a file to get a ready-to-paste link and snippet.

Version

File

Link

Example

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

**Head-to-head tool-calling benchmark** comparing two quantization formats of Qwen3.6-27B on the [tool-eval-bench](https://github.com/SeraphimSerapis/tool-eval-bench/) suite — 84 real-world tool-use scenarios across 15 categories.

| Model | Quantization | Framework | Score |
|---|---|---|---|
| **nvidia/Qwen3.6-27B-NVFP4** 🏆 | FP4 (NVIDIA optimized) | vLLM | **89/100** |
| Unsloth Qwen3.6-27B-UD-Q8\_K\_XL | Q8\_K\_XL GGUF | llama.cpp | **83/100** |

> **Winner: NVFP4 by +6 points.** Stronger on hard scenarios, faster inference, and safer — but GGUF is better at restraint and basic error recovery.

---

## 📊 Results Overview

| Metric | NVFP4 (Winner) | GGUF Q8 | Δ |
|---|---|---|---|
| **Final Score** | **89** /100 | 83 /100 | +6 |
| **Total Points** | 149 / 168 | 139 / 168 | +10 |
| **Quality** | 89 /100 | 83 /100 | +6 |
| **Responsiveness** | **22** /100 | 10 /100 | **+12** |
| **Deployability (α=0.7)** | **69** /100 | 61 /100 | +8 |
| **Median Turn Time** | **7.1s** | 12.7s | −5.6s |
| **Safety Warnings** | **0** | 1 critical | ✅ |

### Category Scores

| Category | NVFP4 | GGUF | Δ |
|---|---|---|---|
| Tool Selection | **100%** | 100% | — |
| Parameter Precision | 100% | 100% | — |
| Multi-Step Chains | **100%** | 75% | **+25** |
| Restraint & Refusal | 83% | **100%** | −17 |
| Error Recovery | 67% | **100%** | −33 |
| Safety & Boundaries | **96%** | 85% | **+11** |
| Toolset Scale | 88% | **100%** | −12 |
| Autonomous Planning | 67% | 67% | — |
| Creative Composition | **83%** | 67% | **+16** |
| Structured Output | 67% | 67% | — |
| **Hard Mode** | **93%** | 70% | **+23** |
| Instruction Following | 80% | 80% | — |
| Context & State | **85%** | 80% | **+5** |

### Performance by Difficulty

| Tier | NVFP4 | GGUF |
|---|---|---|
| Trivial | 75% (3/4) | **100%** (4/4) |
| Easy | 88% (15/17) | **94%** (16/17) |
| Moderate | 84% (26/31) | **87%** (27/31) |
| **Hard** | **75% (18/24)** | 58% (14/24) |
| Very Hard | **75% (6/8)** | 62% (5/8) |

> Hard mode is where NVFP4 pulls away decisively: **75% vs 58%** on hard scenarios and **93% vs 70%** on the adversarial Hard Mode sub-suite.

---

## 🏆 Verdict

**nvidia/Qwen3.6-27B-NVFP4** is the clear winner for production tool-use workloads:

- **Superior hard-mode & multi-step performance** — 93% vs 70% on adversarial scenarios
- **2× faster** — 7.1s median turn time vs 12.7s
- **Stronger safety** — zero safety-critical failures vs 1 (Cross-Turn Sleeper Injection)
- **Perfect repeatability** — 89.0 ± 0.0 across 4 trials
- **Better creative composition** and cross-tool synthesis

> Dismissing speed entirely and looking at **quality alone**: NVFP4 still leads **89 vs 83** (+6). The quality gap is the same as the final score — Responsiveness is a separate axis.

**GGUF Q8\_K\_XL** excels at restraint (100% vs 83%) and basic error recovery (100% vs 67%), but struggles with long-horizon planning, state consistency, and — critically — failed a safety boundary test (TC-60, Cross-Turn Sleeper Injection).

### NVFP4 Known Weaknesses

- Overly eager: calls tools for trivial math it could handle directly
- Weaker on some error recovery scenarios (TC-14 malformed response)
- One structured output schema failure (TC-64) that GGUF passed
- Slightly lower toolset scale score (88% vs 100%)

---

## 🧪 Methodology

Both models were evaluated using [tool-eval-bench v2.0.6](https://github.com/SeraphimSerapis/tool-eval-bench/) on **2026-06-30** under identical conditions:

- **Backend**: OpenAI-compatible `/chat/completions` endpoint
- **Temperature**: 0.6
- **Thinking**: enabled
- **Trials**: 4 runs per model for repeatability
- **Scoring**: 2 pts = pass, 1 pt = partial, 0 pts = fail
- **84 scenarios** across 15 categories (69 base + 15 Hard Mode)

NVFP4 ran on **vLLM** with NVIDIA's FP4-optimized kernel. GGUF Q8\_K\_XL ran on **llama.cpp** with the Unsloth UD variant.

### Exact Command

```bash
tool-eval-bench \
  --seed 42 \
  --trials 8 \
  --hardmode \
  --temperature 0.6 \
  --backend-kwargs '{"chat_template_kwargs": {"enable_thinking": true}}' \
  --base-url http://localhost:8888
```

---

## 🔬 Key Failures

### NVFP4 — Consistent Issues
- **TC-14** — Malformed response: poor error integrity handling
- **TC-45** — `tool_choice=required`: no tool call emitted
- **TC-64** — Simple schema compliance: invalid JSON output
- Partials on restraint (TC-11, 35, 39) and several planning tasks

### GGUF Q8 — Critical Weaknesses
- **TC-60** — Cross-Turn Sleeper Injection ⚠️ **safety-critical failure**
- **TC-56, 61, 63, 72, 74, 75, 80** — Multiple workflow & state failures
- **TC-67, 68** — Structured output & schema violations

---

## 📁 Files

- [`Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4.html`](./Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4.html) — Full interactive comparison report (open in browser)
- This README — Summary of results

---

## ⚙️ Models

| Format | HuggingFace | Notes |
|---|---|---|
| **NVFP4** | [`nvidia/Qwen3.6-27B-NVFP4`](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4) | NVIDIA FP4 quantization, runs on vLLM |
| **GGUF Q8\_K\_XL** | [`unsloth/Qwen3.6-27B-UD-Q8_K_XL`](https://huggingface.co/unsloth/Qwen3.6-27B-UD-Q8_K_XL) | Unsloth UD Q8\_K\_XL GGUF, runs on llama.cpp |

---

## 📝 License

This benchmark report is provided for reference. The compared models are subject to their own licenses (Qwen3.6-27B: Apache 2.0; NVIDIA NVFP4 variant: NVIDIA AI Foundation Models license).

Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4_tools_eval

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Unsloth-Qwen3.6-27B-UD-Q8_K_XL_vs_nvidia-Qwen3.6-27B-NVFP4_tools_eval

Owner Repos

File Explorer

# Use via CDN

Was this content helpful?

Recent Feedback

Download README