ventor_qtest

KO | EN

gitlite — search

#javascript #python #react #typescript #machine-learning #deep-learning #linux #docker #macos #java #windows #pytorch #ios #openai

(★ 33)

No description available.

ventor_qtest 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 🔬 Ventor QTest

[中文文档](README.zh.md)

Quantitative Consistency Testing Framework for LLM Vendors 
 Are you really getting the same model from every provider?

---

## 🤔 The Problem

When you call the same LLM model (e.g., `kimi-k2`) through different API providers — OpenRouter, Groq, Fireworks, Together, etc. — **are you actually getting equivalent outputs?**

Providers may use different quantization levels (FP8, INT4), custom serving stacks, or altered configurations. **Ventor QTest gives you a statistically rigorous answer.**

---

## 🔑 Key Findings

Real experiment results on **Kimi K2** across multiple OpenRouter providers:

| Rank | Provider | mean \|Z\| | vs Official | Verdict |
|:----:|----------|:----------:|:----------:|:-------:|
| 🥇 | **Groq** | 0.519 | −30.3% | ✅ Excellent |
| 🥈 | SiliconFlow Pro | 0.647 | −13.2% | ✅ Good |
| 🥉 | Kimi K2 Turbo | 0.663 | −11.0% | ✅ Good |
| 4 | Novita | 0.720 | −3.4% | ✅ Consistent |
| 📌 | **Kimi K2 Official** | **0.745** | **baseline** | 🏠 Baseline |
| 6 | Fireworks | 0.748 | +0.4% | ✅ Consistent |
| 7 | Together | 0.802 | +7.7% | ⚠️ Slight deviation |
| 8 | DeepInfra | 0.897 | +20.4% | ⚠️ Notable deviation |
| 9 | Atlas Cloud | 1.175 | +57.8% | ❌ Large deviation |
| 10 | SiliconFlow Free | 1.234 | +65.7% | ❌ Large deviation |

> **Key Insight**: Provider consistency varies considerably. Larger deviations may suggest aggressive quantization (e.g., FP8/INT4) or different serving configurations, which *could* affect output quality in some use cases. We recommend running your own tests to evaluate the trade-offs for your specific scenario.

### What does \|Z\| mean?

| \|Z\| Range | Interpretation |
|:-----------:|:--------------:|
| < 0.5 | 🟢 Highly consistent with baseline |
| 0.5 – 1.0 | 🟡 Basically consistent |
| 1.0 – 2.0 | 🟠 Notable deviation — may warrant investigation |
| > 2.0 | 🔴 Large deviation — likely different configuration |

---

## ✨ Features

- 🧮 **Rigorous Statistics** — Z-test based on information-theoretic entropy & variance from logprobs
- 🔄 **Automated Pipeline** — Config → Execute → Aggregate → Visualize, fully automated
- 🌐 **OpenRouter Integration** — Auto-discover all providers for any model with one command
- 📊 **Web Dashboard** — One-click launch, real-time progress, interactive results
- ⚡ **Concurrent & Resilient** — Parallel vendor calls with smart retry on 429/5xx/timeout
- 📋 **Structured Reports** — CSV ranking, per-run details, optional JSON export

---

## 📐 Statistical Methodology

### Core Idea

The question we want to answer: **Does a vendor's output look like it was sampled from the reference model's distribution?**

Traditional approaches compare surface-level text similarity, but that's largely meaningless for LLMs — the same model produces different text every time. What we actually need to compare is the difference at the **probability distribution level**.

### Step-by-Step Walkthrough

Let's walk through a concrete example. Suppose we ask a model to generate a digit sequence, and a vendor returns `1,3,7,2,...`:

**Step 1: Vendor generates a sequence (text only)**

```
Vendor API → "1,3,7,2,9,0,5,4,8,6,..."
```

The vendor only needs to return plain text — no logprobs required.

**Step 2: Query the reference model for probability distributions at each position (requires logprobs + prefix)**

For each character position in the sequence, we use prefix continuation to ask the reference model: "Given the prefix `1,3`, what's your probability distribution for the next character?"

```
Position 0: prefix = ""     → ref model returns p("1")=0.12, p("2")=0.09, p("3")=0.11, ...
Position 1: prefix = "1"    → ref model returns p(",")=0.85, p(".")=0.05, ...
Position 2: prefix = "1,"   → ref model returns p("3")=0.10, p("7")=0.08, ...
...
```

Each position yields a full top_logprobs distribution (typically top 20), with remaining probability mass assigned to an `<OTHER>` bucket and normalized.

**Step 3: Compute statistics at each position**

For each position t, we compute three quantities:

| Quantity | Formula | Meaning |
|----------|---------|---------|
| **Observed loss** | −log p(vendor's actual character) | How "surprising" this character is to the reference model |
| **Expected loss (entropy)** | H(p) = −Σ p(x)·log p(x) | The average loss expected if sampling from the reference distribution |
| **Information variance** | V(p) = E[(−log p(X))²] − H(p)² | The natural fluctuation range of the loss |

Intuition: If the vendor's output truly comes from the reference distribution, then each position's observed loss should be "roughly equal to" the entropy at that position, with fluctuations governed by the variance.

**Step 4: Aggregate into a Z-statistic**

Sum the statistics across all positions:

```
T_obs = Σ observed_losses     (total log-loss of the vendor's sequence)
μ     = Σ H(p_t)              (expected total loss = sum of per-position entropies)
σ     = √(Σ V(p_t))           (std dev = square root of summed per-position variances)

Z = (T_obs − μ) / σ
```

**Why can we simply sum?** Because each position's loss is an independent random variable. By the **Central Limit Theorem**, when the sequence is long enough (typically ≥ 100 characters), T_obs is approximately normally distributed as N(μ, σ²). If the vendor's output truly comes from the reference distribution, Z should approximately follow a standard normal N(0, 1).

### Interpreting the Metrics

**\|Z\|** (absolute Z-score) is the final deviation strength metric:

- \|Z\| ≈ 0 means the vendor's output is exactly as "surprising" as the reference model expects — the probability distributions are highly consistent
- Larger \|Z\| indicates the vendor's output deviates further from the reference at the probability level

A supplementary metric **token_log_dev** (mean per-token log deviation) = \|T_obs − μ\| / sequence_length, provides a variance-independent measure of absolute deviation.

### Why This Works

- **Probability-level comparison, not text-level**: Two models can generate completely different text while having consistent distributions — that's what "same model" fundamentally means
- **Per-character independent testing**: Each position is an independent statistical sample — a 200-character sequence provides 200 data points
- **Central Limit Theorem guarantee**: With sequence lengths ≥ 100, the normal approximation of the Z-statistic is already quite reliable
- **Sensitive to quantization**: FP8/INT4 quantization subtly alters token probability distributions — this method captures these fine differences that surface-level text comparison would miss

---

## 🏗️ Architecture

```text
┌─────────────────────────────────────────────────┐
│                  Config (YAML)                   │
│         questions · vendors · tester · report    │
└──────────────────────┬──────────────────────────┘
                       ▼
┌──────────────────────────────────────────────────┐
│              Orchestrator (runner/)               │
│  ┌────────────┐  ┌────────────┐  ┌────────────┐  │
│  │  Vendor A   │  │  Vendor B   │  │  Vendor C   │ │
│  │ (parallel)  │  │ (parallel)  │  │ (parallel)  │ │
│  └──────┬─────┘  └──────┬─────┘  └──────┬─────┘  │
│         └───────────┬───┘───────────┘             │
│                     ▼                             │
│          ┌──────────────────┐                     │
│          │  check.py Core   │ ← Reference Model   │
│          │  Z-test per char │   (logprobs query)   │
│          └────────┬─────────┘                     │
│                   ▼                               │
│          Per-run JSON files                       │
└──────────────────┬───────────────────────────────┘
                   ▼
┌──────────────────────────────────────────────────┐
│               Aggregator (sum.py)                 │
│  vendors_rank.csv  ·  runs_long.csv  ·  JSON     │
└──────────────────────────────────────────────────┘
                   ▼
┌──────────────────────────────────────────────────┐
│            Web Dashboard (Flask)                  │
│    Setup Page → Progress → Results + Rankings    │
└──────────────────────────────────────────────────┘
```

---

## 🚀 Quick Start

### Prerequisites

```bash
python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
pip install -r requirements.txt
```

### Option 1: Web Interface (Recommended 👈)

```bash
python -m runner.webapp
```

Browser opens automatically → Configure API keys → One-click run → View results.

### Option 2: CLI

```bash
# Edit config/default.yaml with your API keys first
python -m runner.cli run --config config/default.yaml
```

### Option 3: OpenRouter One-Click

```bash
# Discover providers
python -m runner.cli openrouter-providers --model moonshotai/kimi-k2.5

# Auto-discover + test + summarize
python -m runner.cli openrouter-run \
 --openrouter-model moonshotai/kimi-k2.5 \
 --openrouter-api-key <YOUR_KEY>
```

Common flags:
- `--provider-limit 3` — Test only the first 3 providers
- `--include-tags deepinfra,together` — Test specific providers only
- `--exclude-tags siliconflow/fp8` — Exclude specific providers
- `--dry-run` — Preview config without running tests
- `--dump-config config/auto.yaml` — Export auto-generated config

---

## 📁 Project Structure

```text
.
├── check.py                 # Core: logprobs, statistics, hypothesis testing
├── sum.py                   # Aggregator: JSON → vendor ranking CSV
├── main.py                  # Legacy entry point
├── config/
│   └── default.yaml         # Default config (recommended entry)
├── runner/
│   ├── cli.py               # CLI entry: run + auto-summarize
│   ├── orchestrator.py      # Orchestration (vendor calls, persistence, errors)
│   ├── openrouter.py        # OpenRouter API (provider discovery)
│   └── webapp.py            # Flask web app (one-click start)
├── web/                     # Web pages (EN/ZH bilingual)
├── result/                  # Output artifacts (JSON/CSV)
└── log/                     # Test logs
```

## 📊 Output Files

| File | Description |
|------|-------------|
| `result/json/*.json` | Per-run raw results (sequences, per-position stats) |
| `result/summary/vendors_rank.csv` | **Vendor aggregate ranking** (main output) |
| `result/summary/runs_long.csv` | Per-run detail table |

---

## 🧪 GPQA-Diamond benchmark (20 concurrent requests)

`benchmark.performance` pairs repeated Ventor-QTest median D500 with capability
accuracy for the same seven DeepSeek-V4-Flash-0731 routes used by the paper's
Table 1. The default configuration downloads the pinned GPQA-Diamond dataset
and evaluates all 198 items for every route with 20 in-flight requests:

```bash
python -m benchmark.performance --config config/performance_benchmark_gpqa.yaml
```

Keys are read only from the local `.env`. The report contains benchmark accuracy
and its Pearson/Spearman association with the repeated median D500.

---

## ⚙️ Configuration

The default config `config/default.yaml` supports:

| Section | Key Fields |
|---------|-----------|
| `tests[*]` | `questions`, `runs_per_question`, `digits`, `temperature`, `vendors` |
| `tests[*].tester` | Reference model config (`api_key`, `base_url`, `model`, `top_logprobs`) |
| `tests[*].vendors[*]` | `name`, `base_url`, `api_key`, `model`, `schema` (openai/anthropic) |
| `report` | `patterns`, `output_rank_csv`, `output_long_csv`, `sort_by` |

Concurrency tuning:

| Parameter | Default | Description |
|-----------|---------|-------------|
| `vendor_max_workers` | 2 | Concurrent vendor requests per round |
| `max_inflight_ref_requests` | 6 | Max concurrent reference distribution calls |
| `align_tester_temperature` | true | Auto-align tester temp to generation temp |

---

## ❓ FAQ

**Q: What kind of "tests" does this run?**
> It asks LLMs to generate digit sequences, then statistically compares each vendor's output distribution against a reference model using logprob-based Z-tests.

**Q: Does a higher \|Z\| always mean the provider is "bad"?**
> Not necessarily. A higher \|Z\| means the outputs deviate from the reference distribution, which could indicate more aggressive quantization (FP8, INT4), different serving optimizations, or configuration differences. Aggressive quantization *can* degrade quality in some scenarios, but it may also be perfectly acceptable for your use case — lower precision often brings faster inference and lower cost. Use this tool to make an informed trade-off.

**Q: How many runs do I need for reliable results?**
> We recommend at least 4–8 runs per question. More runs = more statistical power.

**Q: Can I test models other than Kimi?**
> Yes! But note that the requirement is **one-sided** — only the **tester** (reference model) has special API requirements:
>
> - Must support `logprobs` and `top_logprobs` in chat completions
> - Must support the **`prefix` field** on assistant messages (used to provide a partial response prefix so the model continues from a specific point and returns the next-token probability distribution)
>
> The **vendors** (models being tested) only need to return plain text via a standard chat completions API — no logprobs or prefix support required. This means you can test virtually any OpenAI-compatible vendor, as long as your chosen reference model meets the above two requirements.

**Q: Why not use KL divergence to compare distributions?**
> KL divergence (D_KL(P \|\| Q)) requires **full probability distributions from both sides** — both P (vendor) and Q (reference). In our scenario, the vendor API typically only returns plain text, not logprobs. We only have the probability distribution from **one side** (the reference model). Our Z-test approach is designed precisely for this one-sided constraint: it uses the reference model's distribution to evaluate how "surprising" the vendor's text output is, without ever needing the vendor's own probability distribution. This makes the method applicable to virtually any vendor API.

---

## 🔒 Security

- **Never** commit real API keys to Git — use environment variables
- Share `result/summary/*.csv` instead of raw JSONs to avoid exposing request details
- See `.env.example` for environment variable patterns

---

## 📜 License

MIT

---

Built with 🔬 statistics and ☕ curiosity

ventor_qtest

ventor_qtest 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

ventor_qtest

ventor_qtest 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README