docs: Qwen3.8 spec-matrix benchmark page; add matrix tools; final env n_max=5
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not 10), the ubatch findings (256 is the safe/fast point), and the premature-stop sampling-variance investigation.
This commit is contained in:
1 parent
a818be70dc
commit
0eda208705
4 files changed
+235
No files matched your search
@@ -52,6 +52,7 @@ MTP-ONLY draft, port 7770]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B UD-Q8_K_XL bundled-MTP spec matrix; winner draft-mtp n_max=5
|
||||
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
|
||||
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service on Dankclaw
|
||||
(`frank-default-large`).
|
||||
|
||||
## Result (applied)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
|
||||
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
|
||||
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
|
||||
2 concurrent streams. No premature stop.
|
||||
|
||||
## How MTP works on Qwen3.8
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model`.
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
||||
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
||||
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
||||
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
||||
analysis is `scripts/analyze-spec-matrix.py`.
|
||||
|
||||
## Matrix Caveat: Benchmark Prompt
|
||||
|
||||
The matrix used a repetitive 1100-token prompt
|
||||
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
||||
perfectly, which inflated the numbers for long drafts. The matrix reported
|
||||
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
||||
different results:
|
||||
|
||||
| n_max | median t/s (realistic) |
|
||||
| ----- | ----------------------- |
|
||||
| 4 | 12.5 |
|
||||
| 5 | 13.3 (peak) |
|
||||
| 6 | 10.6 |
|
||||
| 7 | 10.1 |
|
||||
| 10 | 8.6 |
|
||||
| none (baseline) | 7.2 |
|
||||
|
||||
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
||||
acceptance against draft cost.
|
||||
|
||||
## Premature Stop Investigation
|
||||
|
||||
Two failure patterns appeared during early production tests:
|
||||
|
||||
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
||||
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
||||
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
||||
both: ~30 t/s single, concurrent-safe.
|
||||
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
||||
sometimes stops after 1-2 tokens because the model legitimately emits an
|
||||
EOS on a wall of repeated text. This is sampling variance, not a
|
||||
corruption bug. Realistic prompts never stop early.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
`scripts/frank-default-large.env`:
|
||||
|
||||
```ini
|
||||
SPEC_TYPE=draft-mtp
|
||||
SPEC_DRAFT_N_MAX=5
|
||||
SPEC_DRAFT_P_MIN=
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2
|
||||
```
|
||||
|
||||
Window: 524288 total / 2 = 262144 per slot, matching the pi
|
||||
`models.json` contextWindow for `frank-default-large`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-large-benchmark]]
|
||||
- [[ling-3.0-flash-trial]]
|
||||
- [[deployment]]
|
||||
Reference in new issue
Block a user