Frank
04f4e2c6d4
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
142 lines
6.3 KiB
Markdown
142 lines
6.3 KiB
Markdown
---
|
|
type: reference
|
|
description: "This page records the speculative-decoding benchmark history and the"
|
|
tags:
|
|
- llm
|
|
- speculative
|
|
timestamp: 2026-09-03
|
|
---
|
|
|
|
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
|
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
|
> [deployment.md](deployment.md) for the active configuration.
|
|
|
|
# Qwen3.8-27B Speculative-Decoding Matrix
|
|
|
|
This page records the speculative-decoding benchmark history and the
|
|
production tuning for the then-Qwen3.8-27B service (`frank-dragoon`), at the
|
|
time a pipeline-parallel head with a Dankserver worker.
|
|
|
|
## Production state (2026-08-17)
|
|
|
|
The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone
|
|
Q8_0 draft on ROCm0 only, `--tensor-split 0.68,0.32` (45 of 65 blocks), K `q8_0`
|
|
/ V `f16`, pool 262144 at one slot, worker steady at 23854/24576 MiB — is
|
|
recorded once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Production
|
|
state, together with the split-limit arithmetic (why 45 blocks is the maximum),
|
|
the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That
|
|
era's endpoint identity is an [[frank-dragoon]] era. This page keeps only the
|
|
speculative-decoding measurements.
|
|
|
|
## How MTP works on Qwen3.8 (bundled-head era)
|
|
|
|
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
|
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
|
context from the target model. Do not pass `--spec-draft-model` for those
|
|
files. This is superseded: the production target of 2026-08-17 carried no
|
|
bundled head, and the serve script passed the standalone draft sidecar with
|
|
`--spec-draft-model` (shape: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]).
|
|
|
|
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
|
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
|
|
|
## History (superseded configurations, 2026-08-14 to 08-16)
|
|
|
|
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
|
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
|
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
|
standalone configuration only; the PP head runs ROCm0.
|
|
|
|
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
|
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
|
concurrent streams. No premature stop.
|
|
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
|
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
|
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
|
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
|
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
|
block-diffusion machinery, which shares one draft context across
|
|
parallel slots; concurrent block drafting faults in HIP stream sync.
|
|
This is the same multi-slot bug class that retired `draft-dflash` (premature
|
|
EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a
|
|
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests
|
|
queue safely). Retired the same day.
|
|
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
|
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
|
test passes). Chosen for parallel safety after DSpark; kept as the
|
|
fallback under MTP in the 2026-08-17 shape.
|
|
|
|
## Benchmark Method
|
|
|
|
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
|
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
|
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
|
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
|
analysis is `scripts/analyze-spec-matrix.py`.
|
|
|
|
## Matrix Caveat: Benchmark Prompt
|
|
|
|
The matrix used a repetitive 1100-token prompt
|
|
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
|
perfectly, which inflated the numbers for long drafts. The matrix reported
|
|
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
|
different results:
|
|
|
|
| n_max | median t/s (realistic) |
|
|
| ----- | ----------------------- |
|
|
| 4 | 12.5 |
|
|
| 5 | 13.3 (peak) |
|
|
| 6 | 10.6 |
|
|
| 7 | 10.1 |
|
|
| 10 | 8.6 |
|
|
| none (baseline) | 7.2 |
|
|
|
|
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
|
acceptance against draft cost.
|
|
|
|
## Premature Stop Investigation
|
|
|
|
Two failure patterns appeared during early production tests:
|
|
|
|
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
|
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
|
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
|
both: ~30 t/s single, concurrent-safe.
|
|
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
|
sometimes stops after 1-2 tokens because the model legitimately emits an
|
|
EOS on a wall of repeated text. This is sampling variance, not a
|
|
corruption bug. Realistic prompts never stop early.
|
|
|
|
## High-Context Backend Comparison (2026-08-15)
|
|
|
|
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
|
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
|
ubatch 256, single slot, on scratch port 7892.
|
|
|
|
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
|
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
|
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
|
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
|
|
|
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
|
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
|
(users stream tokens). This verdict applies to the standalone configuration; the PP
|
|
head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp
|
|
PR #23897 workaround) keeps the first backend's device.
|
|
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
|
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
|
that is the long-context attention penalty, not a spec artifact.
|
|
|
|
## Where the tunables live
|
|
|
|
That era's head and worker env/units are listed (and marked deleted) in
|
|
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Files. `scripts/frank-dragoon.env`
|
|
is no longer a PP-head env at all — it is the halogen-flash-server trio of the
|
|
current era, which [[frank-dragoon]] owns.
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon]]
|
|
- [[ling-3.0-flash-trial]]
|
|
- [[deployment]]
|