docs(vault): strip restated fleet facts from 17 technique pages
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps and verdicts; per-endpoint shapes, digests and era measurements are cited to the endpoint page that owns them. Also repairs the four pointers the pass left empty (each verified against the cited owner before landing): - EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]] - 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]] - bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]] - llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]] Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a hard-wrapped [[Dankserver System Specs]] link that failed link integrity. Orphan census over the diff: 0 numbers lost (comma-insensitive match against the whole vault plus AGENTS.md).
This commit is contained in:
1 parent
eb21dbc705
commit
04f4e2c6d4
18 files changed
+348
-425
No files matched your search
@@ -14,54 +14,28 @@ timestamp: 2026-09-03
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-dragoon`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
production tuning for the then-Qwen3.8-27B service (`frank-dragoon`), at the
|
||||
time a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
## Production state (2026-08-17)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-17, load 02:06 + smoke tests). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
|
||||
imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
|
||||
standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
|
||||
(1718 MiB, zero worker impact).
|
||||
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
|
||||
`--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
|
||||
0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
|
||||
stronger than the old bundled head (0.739/0.554/0.427/0.340).
|
||||
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
|
||||
784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
|
||||
3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
|
||||
enabled with the mismatch** (`resolve_fused_ops: Flash Attention
|
||||
enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
|
||||
at 9b05354ec — see the PP doc.
|
||||
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
|
||||
3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
|
||||
margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
|
||||
~0.85 GiB. Boundary math: the PP doc.
|
||||
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
|
||||
`models.json` declares 262144 (contextWindow = maxTokens, house
|
||||
convention).
|
||||
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
|
||||
(tight but stable; all allocations happen at load).
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
The pipeline-parallel shape this matrix was tuned for — Q6_K target, standalone
|
||||
Q8_0 draft on ROCm0 only, `--tensor-split 0.68,0.32` (45 of 65 blocks), K `q8_0`
|
||||
/ V `f16`, pool 262144 at one slot, worker steady at 23854/24576 MiB — is
|
||||
recorded once, in [[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Production
|
||||
state, together with the split-limit arithmetic (why 45 blocks is the maximum),
|
||||
the KV-per-layer bill and the FA-all-quants rebuild the mismatch needed. That
|
||||
era's endpoint identity is an [[frank-dragoon]] era. This page keeps only the
|
||||
speculative-decoding measurements.
|
||||
|
||||
## How MTP works on Qwen3.8 (bundled-head era)
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model` for those
|
||||
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
|
||||
has NO bundled head, and the serve script passes the standalone draft
|
||||
sidecar with `--spec-draft-model` (see Production state above).
|
||||
files. This is superseded: the production target of 2026-08-17 carried no
|
||||
bundled head, and the serve script passed the standalone draft sidecar with
|
||||
`--spec-draft-model` (shape: [[Pipeline Parallelism/Qwen3.8 PP Deployment]]).
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
@@ -83,14 +57,14 @@ standalone configuration only; the PP head runs ROCm0.
|
||||
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
||||
block-diffusion machinery, which shares one draft context across
|
||||
parallel slots; concurrent block drafting faults in HIP stream sync.
|
||||
This is the same multi-slot bug class that retired `draft-dflash`
|
||||
(premature EOS). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
|
||||
(requests queue safely). Retired the same day.
|
||||
This is the same multi-slot bug class that retired `draft-dflash` (premature
|
||||
EOS, retired 2026-08-14). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 (requests
|
||||
queue safely). Retired the same day.
|
||||
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
||||
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
||||
test passes). Chosen for parallel safety after DSpark; kept as the
|
||||
fallback under MTP in the current config.
|
||||
fallback under MTP in the 2026-08-17 shape.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
@@ -146,19 +120,19 @@ ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens). This verdict applies to the standalone
|
||||
configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
|
||||
(users stream tokens). This verdict applies to the standalone configuration; the PP
|
||||
head runs ROCm0 — HIP registers before Vulkan and the iGPU dedup rule (the llama.cpp
|
||||
PR #23897 workaround) keeps the first backend's device.
|
||||
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
## Where the tunables live
|
||||
|
||||
The live tunables live in `scripts/frank-dragoon.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
q8_0/f16 KV.
|
||||
That era's head and worker env/units are listed (and marked deleted) in
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]] §Files. `scripts/frank-dragoon.env`
|
||||
is no longer a PP-head env at all — it is the halogen-flash-server trio of the
|
||||
current era, which [[frank-dragoon]] owns.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in new issue
Block a user