Files

T

Frank 04f4e2c6d4 docs(vault): strip restated fleet facts from 17 technique pages

Enforces "one fact, one owner": technique pages keep mechanism, laws, traps
and verdicts; per-endpoint shapes, digests and era measurements are cited to
the endpoint page that owns them.

Also repairs the four pointers the pass left empty (each verified against the
cited owner before landing):
- EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]]
- 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]]
- bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]]
- llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]]

Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the
owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a
hard-wrapped [[Dankserver System Specs]] link that failed link integrity.

Orphan census over the diff: 0 numbers lost (comma-insensitive match against
the whole vault plus AGENTS.md).

2026-10-19 01:33:47 -07:00

6.3 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Qwen3.8-27B Speculative-Decoding Matrix

This page records the speculative-decoding benchmark history and the production tuning for the then-Qwen3.8-27B service (frank-dragoon), at the time a pipeline-parallel head with a Dankserver worker.

Production state (2026-08-17)

The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone Q8_0 draft on ROCm0 only, --tensor-split 0.68,0.32 (45 of 65 blocks), K q8_0 / V f16, pool 262144 at one slot, worker steady at 23854/24576 MiB — is recorded once, in Pipeline Parallelism/Qwen3.8 PP Deployment §Production state, together with the split-limit arithmetic (why 45 blocks is the maximum), the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That era's endpoint identity is an frank-dragoon era. This page keeps only the speculative-decoding measurements.

How MTP works on Qwen3.8 (bundled-head era)

The unsloth UD-* GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft context from the target model. Do not pass --spec-draft-model for those files. This is superseded: the production target of 2026-08-17 carried no bundled head, and the serve script passed the standalone draft sidecar with --spec-draft-model (shape: Pipeline Parallelism/Qwen3.8 PP Deployment).

The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (9b05354ec) with GGML_HIP + gfx1151 + Vulkan.

History (superseded configurations, 2026-08-14 to 08-16)

All superseded configs used Qwen3.8-27B-UD-Q8_K_XL.gguf (29.29 GiB, bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the fallback layer under MTP. The Vulkan0 verdict from this era applies to the standalone configuration only; the PP head runs ROCm0.

2026-08-14: bundled MTP — draft-mtp, n_max 5, parallel 2, ctx 524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2 concurrent streams. No premature stop.
2026-08-15: DSpark sidecar — magnitudedev/Qwen3.8-27B-DSpark-GGUF (Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k prose prompts crashed the ROCm backend with ROCm error: an illegal memory access in common_speculative_impl_draft_dflash::draft (status 6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash block-diffusion machinery, which shares one draft context across parallel slots; concurrent block drafting faults in HIP stream sync. This is the same multi-slot bug class that retired draft-dflash (premature EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a 67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests queue safely). Retired the same day.
2026-08-15 to 08-16: --spec-default (ngram-mod), Vulkan0 — n_match 24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream test passes). Chosen for parallel safety after DSpark; kept as the fallback under MTP in the 2026-08-17 shape.

Benchmark Method

The matrix ran 54 configurations: 3 spec types (draft-mtp,ngram-mod, draft-mtp, ngram-mod) x --spec-draft-n-max 2..10 x --spec-draft-p-min {off, 0.8}. Each config loaded llama-server, then ran 3 generations. The tool is scripts/spec-matrix-bench.sh; analysis is scripts/analyze-spec-matrix.py.

Matrix Caveat: Benchmark Prompt

The matrix used a repetitive 1100-token prompt (scripts/bench_live.sh style). MTP drafts that repeated sentence almost perfectly, which inflated the numbers for long drafts. The matrix reported draft-mtp n_max=10 at ~30 t/s. A realistic story prompt gave very different results:

n_max	median t/s (realistic)
4	12.5
5	13.3 (peak)
6	10.6
7	10.1
10	8.6
none (baseline)	7.2

Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances acceptance against draft cost.

Premature Stop Investigation

Two failure patterns appeared during early production tests:

Speed collapse. ubatch >= 512 reduced MTP acceptance to ~25% and generation to ~9 t/s. ubatch 128 with batch >= 2048 triggered a premature instant-EOS under 2 concurrent slots. ubatch 256 avoids both: ~30 t/s single, concurrent-safe.
Apparent premature-EOS. The repetitive benchmark prompt with temp 1.0 sometimes stops after 1-2 tokens because the model legitimately emits an EOS on a wall of repeated text. This is sampling variance, not a corruption bug. Realistic prompts never stop early.

High-Context Backend Comparison (2026-08-15)

Tool: scripts/bench-high-ctx.sh <device> <ctx>. Both runs at ctx 262144 with a ~200k-token realistic-prose prompt, --spec-default, batch 8192 / ubatch 256, single slot, on scratch port 7892.

Device	Prompt tokens	Prefill (t/s)	Generation (t/s)	Wall (single req)
ROCm0	202822	119.1	3.83	1770 s
Vulkan0	200235	106.3	5.68	1929 s

Verdict: at very high context, Vulkan0 decodes 48% faster (5.68 vs 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost (users stream tokens). This verdict applies to the standalone configuration; the PP head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp PR #23897 workaround) keeps the first backend's device. Both tg_3s values were rock-steady across the run (3.83 and 5.67 t/s). High-context decode collapses on both backends versus ~25 t/s at 16k ctx; that is the long-context attention penalty, not a spec artifact.

Where the tunables live

That era's head and worker env/units are listed (and marked deleted) in Pipeline Parallelism/Qwen3.8 PP Deployment §Files. scripts/frank-dragoon.env is no longer a PP-head env at all — it is the halogen-flash-server trio of the current era, which frank-dragoon owns.

6.3 KiB Raw Blame History