From f917a2ef5532aff8d927e145230db6a65b68e8a9 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 21:01:45 -0700 Subject: [PATCH] docs: the runbook records the DSpark excursion as an excursion The note described DSpark as the deployed drafter and 40:25 as the split; both were true for a few hours. It now leads with DFlash2 at 36:29 and keeps the excursion as the record it is: - the drafter comparison table with DSpark's REAL size (1,455,376,576, not the 1,359,284,737 that went into the fetch script), and why it lost: no sliding window, so its draft KV spanned the whole context. - the split's two moves as one story: the batch pushed it to 40:25, the drafter swap pulled it back to 36:29, with both measurement tables. - the layer-size law (0.43 GiB DeltaNet, 1.43 GiB attention) as the reason the balance is measured and not interpolated. - the verified-behaviour section now carries the LIVE numbers: 4.71 / 4.91 GiB free (4.43 / 4.61 under load, 0.18 GiB gap), the DFlash2 init lines, and draft acceptance from both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0). - the fault note scopes its own evidence: 39:26 was loaded, not exercised. --- .../LLM/frank-medium-llamacpp-dflash2.md | 159 +++++++++++------- 1 file changed, 97 insertions(+), 62 deletions(-) diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index e578fd8..c687ac5 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,11 +1,12 @@ --- type: configuration -title: frank-medium llama.cpp Runbook (DSpark drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards, 262144 ctx over two slots. +title: frank-medium llama.cpp Runbook (DFlash2 drafter) +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, 262144 ctx over two slots. tags: - frank-medium - dankserver - llama-cpp + - dflash2 - dspark - qwen35 - gfx1201 @@ -14,13 +15,13 @@ tags: timestamp: 2026-09-16 --- -# frank-medium — llama.cpp (DSpark drafter) +# frank-medium — llama.cpp (DFlash2 drafter) The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp** serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL` -with the **magnitudedev DSpark** block-diffusion drafter (it replaced z-lab -DFlash2 the same day — see the swap section below), as **one process across -both Radeon AI PRO R9700 cards**, layer-split. +with the **z-lab DFlash2** block-diffusion drafter, as **one process across both +Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's +place for a few hours the same day and reverted — see *The DSpark excursion*. It replaced the vllm-radiance MXFP4 replica pair + routing proxy ([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in the same day the pair went live. It is also the box's **only** LLM runtime: the @@ -102,7 +103,7 @@ bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc) | File | Bytes | Role | |---|---:|---| | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB | -| `Qwen3.8-27B-DSpark-Q8_0.gguf` | 1,359,284,737 | DSpark drafter, 1.27 GiB | +| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB | Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified — a truncated 31 GB GGUF is otherwise a load-time crash). @@ -114,12 +115,12 @@ vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image endpoint is frank-small. **The file also carries the model's MTP head** (`blk.64.nextn.*`). Under -`--spec-type draft-dspark` those tensors are deliberately **not** loaded — the +`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` — -because DSpark drafts from its own 5-layer sidecar. Switching (or chaining) -`--spec-type draft-mtp,draft-dspark` is a one-line env change that pulls the -head in; it is not enabled because the DSpark path is the measured one and -because a chain cannot hold two context-backed drafters at all (below). +because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining) +`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the +head in; it is not enabled because the DFlash2 path is the measured one (below) +and because a chain cannot hold two context-backed drafters at all. ## Deployment @@ -143,31 +144,38 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `40,25` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter (see the swap section) | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | | `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context | | `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance | | `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | -| `--spec-type` / `--spec-draft-n-max` | `draft-dspark,ngram-mod` / `6` | DSpark blocks are 7 tokens, so 6 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer | +| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer | | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | | `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint | | `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited | | `--no-mmproj` | — | text only | -## The DSpark swap (2026-09-16, hours after the cutover) +## The DSpark excursion (2026-09-16, tried and reverted) -The endpoint went live on **z-lab DFlash2** and was switched to -**magnitudedev/Qwen3.8-27B-DSpark** the same day. Both are 5-layer -block-diffusion sidecars, but they differ in ways that decide the deployment: +The endpoint went live on **z-lab DFlash2**, was switched to +**magnitudedev/Qwen3.8-27B-DSpark** for a few hours the same day, and was put +back. Both are 5-layer block-diffusion sidecars, and their differences are what +decided it: -| | DFlash2 (before) | DSpark (deployed) | +| | DFlash2 (deployed) | DSpark (tried, reverted) | |---|---|---| -| file | 2,056,414,816 B — 1.92 GiB | 1,359,284,737 B — 1.27 GiB | +| file | 2,056,414,816 B — 1.92 GiB | 1,455,376,576 B — 1.36 GiB | | draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) | | attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector | | draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context | +> **A fetch-script bug this threw up.** The DSpark file's size was recorded as +> 1,359,284,737 when it was fetched — 96 MB short of the Hub's 1,455,376,576 — +> and that wrong number became the fetch script's declared size, so +> `--verify-only` failed on a payload that was in fact byte-correct. Both +> drafters' sha256 have since been checked against the Hub's LFS oids and match. + **The draft context is sized from the target's.** `cparams.n_ctx = llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it independently, so a drafter that declares no sliding window caches the entire @@ -185,36 +193,59 @@ Read `sliding_window` out of a drafter's GGUF header before sizing a spec-decode deployment — 40 lines of `struct` parsing is enough, and the flag-level configuration looks identical either way. -### The split then had to move again for the batch +### The split moved twice: once for the batch, once for the drafter -Tripling the batch (2048/512 -> 6144/1536) made the two cards diverge: at -262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against ROCm0's -~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs — which -left the draft-carrying card at **0.08 GiB free** against ROCm0's 3.78 under -the 38:27 split that had been balanced at the smaller batch. One `-ts` step is -worth ~1.5 GiB at this size, so the balance was found by measurement, not -arithmetic: +**Under DSpark.** Tripling the batch (2048/512 -> 6144/1536) made the two cards +diverge: at 262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against +ROCm0's ~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs — +which left the draft-carrying card at 0.08 GiB free against ROCm0's 3.78 under +the 38:27 split that had been balanced at the smaller batch: | `-ts` | GPU 0 free | GPU 1 free | |---|---:|---:| | 38,27 | 3.78 GiB | 0.08 GiB | | 41,24 | 1.47 GiB | 2.38 GiB | -| **40,25 — deployed** | **1.88 GiB** | **1.88 GiB** | +| 40,25 | 1.88 GiB | 1.88 GiB | -The balance is also what **maximises the context**: an unbalanced split has to -buy its headroom out of `CTX_SIZE`, and `CTX_SIZE` is already at the model's -trained window. The operator's floor for this configuration is 1.5 GiB per -card, and `scripts/frank-medium.env`'s sizing table carries the measurements -that 8192/2048 does *not* fit here at all (its 2020 MiB pp buffer fails to -allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB. +**Then DSpark came off**, which frees ~4.3 GiB on ROCm1 (5.0 GiB of draft KV +out, 0.65 GiB of heavier DFlash2 weights in) and moves the balance the other +way. Re-measured at the same batch, the same two slots and the same window: + +| `-ts` | GPU 0 free | GPU 1 free | gap | +|---|---:|---:|---:| +| 39,26 | 2.42 GiB | 7.20 GiB | 4.78 | +| 38,27 | 3.85 GiB | 5.77 GiB | 1.92 | +| 37,28 | 4.28 GiB | 5.34 GiB | 1.06 | +| **36,29 — deployed** | **4.71 GiB** | **4.91 GiB** | **0.20** | +| 35,30 | 5.14 GiB | 4.48 GiB | 0.66 | +| 34,31 | 6.57 GiB | 3.05 GiB | 3.52 | + +**One `-ts` step is one layer, and this stack's layers are not the same size.** +qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth +**~0.43 GiB** on a DeltaNet boundary and **~1.43 GiB** on a full-attention one +(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144). Both fell out +of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an +attention layer. Balance by measurement, never by interpolating. + +36:29 is both the minimum and the point that leaves the most room: it clears the +operator's 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought +out of `CTX_SIZE` and the model's full 262144-token window stays. + +A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not +be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure, +not a total-bytes one, and it has **not** been re-measured with DFlash2's ~4.9 +GiB of headroom. (`tests/frank_medium_unit.bats` fits the deployed point and +predicts it still would not fit, but that is an extrapolation of a coefficient +fitted at ubatch 1536.) > **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its > eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the > engine surfaced `ROCm error: an illegal memory access was encountered` -> > `ggml_abort` — a fault in the cross-card boundary copy, which is this layer -> split's hot path. It has not reproduced (38:27 served the whole e2e gate and -> a burst of requests; 40:25 has served the verification battery and a deploy -> gate), and it is **not** an OOM: no amount of headroom would prevent it. +> split's hot path. It has not reproduced (36:29 has since served the whole +> verification battery and the e2e gate; 39:26 itself was only *loaded* at the +> deployed batch, not exercised), and it is **not** an OOM: no amount of +> headroom would prevent it. ## Verified serving behaviour (2026-09-16) @@ -222,24 +253,30 @@ allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB. `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s. - Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports `modalities: {vision: false}` and the sampler defaults above. -- DSpark engaged — the init lines are the proof, not the flags: +- DFlash2 engaged — the init lines are the proof, not the flags: ``` - common_speculative_init_result: loading draft model '.../Qwen3.8-27B-DSpark-Q8_0.gguf' - common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark' - common_speculative_impl_draft_dflash: - n_max=6, n_min=0, p_min=0.00 - common_speculative_impl_draft_dflash: - block_size=7, mask_token_id=248077, n_extract=5, sample_from_anchor=true + common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf' + common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash' + common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00 + common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true ``` - `block_size=7` and `mask_token_id=248077` are DSpark's own metadata. The - implementation still logs under the `draft_dflash` symbol because dspark *is* - that code path plus the Markov head — the loader prefers a dspark sidecar - when one is present. + `block_size=8` and `n_extract=5` are DFlash2's own metadata; the candidate + selector is what distinguishes it from DFlash 1. DSpark logs under the same + `draft_dflash` symbol with `block_size=7` and a Markov head, so read the + numbers, not the implementation name. - Both cards hold the model and they are **balanced**: at 262144 with the - deployed batch, `rocm-smi` reports **29.98 GiB used on each card** of - 31.86 GiB — 1.88 GiB free on each, measured with two concurrent 1832-token - prompts in flight. The 38:27 split that preceded the batch change was - lopsided (3.78 / 0.08 GiB free); see the swap section above. + deployed batch and split, `rocm-smi` reports **4.71 GiB free on GPU 0 and + 4.91 GiB on GPU 1** of 31.86 GiB each — the same figures the balance search + predicted to 0.01 GiB. Under load (two concurrent requests, one per slot) they + sit at 4.43 / 4.61, a 0.18 GiB gap, and both are far above the operator's + 1.5 GiB floor. +- Drafting is live on **both slots**, which is what a 2-slot deployment has to + prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted / + 70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and + `0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair. + Acceptance moves with the text; the point is that neither slot is idle. - **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured through a *single* slot, at the previous configuration: the raised window is real rather than declared, and it is 22 % beyond the 114688 the vLLM @@ -281,12 +318,11 @@ no proxy. ### What the drafter is worth — and why MTP is not the alternative -Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter: -four configurations of the same binary, identical prompts at `temperature 0`, -unit stopped between runs (one host, one copy of a 29 GiB model). 1k-token -prompt, 512-token budget. These are the numbers that selected a trained -multi-token drafter over the model's own MTP head; DSpark replaced DFlash2 for -the reasons in the swap section, and it is that conclusion which carried over. +Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter — +the one deployed now: four configurations of the same binary, identical prompts +at `temperature 0`, unit stopped between runs (one host, one copy of a 29 GiB +model). 1k-token prompt, 512-token budget. These are the numbers that selected a +trained multi-token drafter over the model's own MTP head. | Configuration | Decode | Drafting | VRAM (card 1) | |---|---:|---|---:| @@ -323,10 +359,9 @@ the drafter over one. (MTP is the opposite: it *is* context-backed, which is why `draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types first, then the drafters — also means CLI order is irrelevant. -The measurements below are DFlash2's; the chain is now `draft-dspark,ngram-mod`. -The matcher's behaviour — zero drafts on prose, 100 % on echo — is a property -of the matcher, not of the drafter beside it, but re-run -`scripts/bench-spec-ab.py` if the DSpark numbers themselves matter. +The measurements below are DFlash2's — the deployed configuration. (During the +DSpark excursion the chain was `draft-dspark,ngram-mod`; the matcher's behaviour +is a property of the matcher, not of the drafter beside it.) Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a proposer can and cannot match: @@ -429,7 +464,7 @@ gets the server defaults above; a client that sends its own wins. Text only. llama-server starts, and a drafter whose GGUF the build does not recognise fails the load instead. - **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is - correct for `draft-dspark`. It is not a truncated download: + correct for `draft-dflash`. It is not a truncated download: `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes. - **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a gfx1200 sibling binary links, loads the model and then segfaults on these