--- type: reference description: "This page records the speculative-decoding benchmark history and the" tags: - llm - speculative timestamp: 2026-09-03 --- > **SUPERSEDED 2026-08-17.** The Frank redo retired this topology. > See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and > [deployment.md](deployment.md) for the active configuration. # Qwen3.8-27B Speculative-Decoding Matrix This page records the speculative-decoding benchmark history and the production tuning for the then-Qwen3.8-27B service (`frank-dragoon`), at the time a pipeline-parallel head with a Dankserver worker. ## Production state (2026-08-17) The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone Q8_0 draft on ROCm0 only, `--tensor-split 0.68,0.32` (45 of 65 blocks), K `q8_0` / V `f16`, pool 262144 at one slot, worker steady at 23854/24576 MiB — is recorded once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Production state, together with the split-limit arithmetic (why 45 blocks is the maximum), the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That era's endpoint identity is an [[frank-dragoon]] era. This page keeps only the speculative-decoding measurements. ## How MTP works on Qwen3.8 (bundled-head era) The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft context from the target model. Do not pass `--spec-draft-model` for those files. This is superseded: the production target of 2026-08-17 carried no bundled head, and the serve script passed the standalone draft sidecar with `--spec-draft-model` (shape: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]). The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan. ## History (superseded configurations, 2026-08-14 to 08-16) All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the fallback layer under MTP. The Vulkan0 verdict from this era applies to the standalone configuration only; the PP head runs ROCm0. - **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx 524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop. - **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF` (Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` in `common_speculative_impl_draft_dflash::draft` (status 6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash block-diffusion machinery, which shares one draft context across parallel slots; concurrent block drafting faults in HIP stream sync. This is the same multi-slot bug class that retired `draft-dflash` (premature EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a 67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests queue safely). Retired the same day. - **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match 24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream test passes). Chosen for parallel safety after DSpark; kept as the fallback under MTP in the 2026-08-17 shape. ## Benchmark Method The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`, `draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x `--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`, then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`; analysis is `scripts/analyze-spec-matrix.py`. ## Matrix Caveat: Benchmark Prompt The matrix used a repetitive 1100-token prompt (`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost perfectly, which inflated the numbers for long drafts. The matrix reported `draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very different results: | n_max | median t/s (realistic) | | ----- | ----------------------- | | 4 | 12.5 | | 5 | 13.3 (peak) | | 6 | 10.6 | | 7 | 10.1 | | 10 | 8.6 | | none (baseline) | 7.2 | Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost. ## Premature Stop Investigation Two failure patterns appeared during early production tests: 1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids both: ~30 t/s single, concurrent-safe. 2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0` sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early. ## High-Context Backend Comparison (2026-08-15) Tool: `scripts/bench-high-ctx.sh `. Both runs at ctx 262144 with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 / ubatch 256, single slot, on scratch port 7892. | Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) | | ------ | ------------- | ------------- | ---------------- | ----------------- | | ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** | | Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s | Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost (users stream tokens). This verdict applies to the standalone configuration; the PP head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp PR #23897 workaround) keeps the first backend's device. Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s). High-context decode collapses on both backends versus ~25 t/s at 16k ctx; that is the long-context attention penalty, not a spec artifact. ## Where the tunables live That era's head and worker env/units are listed (and marked deleted) in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Files. `scripts/frank-dragoon.env` is no longer a PP-head env at all — it is the halogen-flash-server trio of the current era, which [[frank-dragoon]] owns. ## See Also - [[frank-dragoon]] - [[ling-3.0-flash-trial]] - [[deployment]]