From 718a283afc28a9bb946787bc89bd42cdf6668d12 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 20:21:31 -0700 Subject: [PATCH] docs: the frank-medium runbook catches up with DSpark, 2 slots and the rebalance The note still described DFlash2, one slot at 262144 and a 1:1 split -- all of which the day's work replaced. It now carries: - the DSpark swap with the table that matters (1.27 vs 1.92 GiB, block_size 7 vs 8 so n_max 6 vs 7, a Markov head instead of the candidate selector, and NO sliding window) and the rule behind the sizing: the draft context is built with n_ctx = llama_n_ctx(ctx_tgt), so a drafter without a window caches the whole target context -- 5.0 GiB here, against DFlash2's ~40 MiB. - the failure that rule produces when ignored: load succeeds, /health answers, and the first draft decode dies with an OOM at 1:1. - why the split moved a second time: tripling the batch costs ROCm1 about twice what it costs ROCm0, with the 38,27 -> 41,24 -> 40,25 measurement table. - the deployed flags, the 2-slot shape (n_slots = 2, n_ctx_slot = 131072), the DSpark init lines, and balanced 29.98 / 29.98 GiB cards. - honest scoping on the older measurements: the throughput, drafter-worth and ngram tables were taken on DFlash2, and the agent-loop numbers on one slot. - the 39:26 page-fault signature as a thing to watch, marked as NOT an OOM. --- .../LLM/frank-medium-llamacpp-dflash2.md | 175 +++++++++++++----- 1 file changed, 126 insertions(+), 49 deletions(-) diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 60710d2..e578fd8 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,12 +1,12 @@ --- type: configuration -title: frank-medium llama.cpp + DFlash2 Runbook -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards. +title: frank-medium llama.cpp Runbook (DSpark drafter) +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards, 262144 ctx over two slots. tags: - frank-medium - dankserver - llama-cpp - - dflash2 + - dspark - qwen35 - gfx1201 - r9700 @@ -14,13 +14,13 @@ tags: timestamp: 2026-09-16 --- -# frank-medium — llama.cpp + DFlash2 +# frank-medium — llama.cpp (DSpark drafter) The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp** serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL` -with the **z-lab DFlash2** block-diffusion drafter, as **one process across +with the **magnitudedev DSpark** block-diffusion drafter (it replaced z-lab +DFlash2 the same day — see the swap section below), as **one process across both Radeon AI PRO R9700 cards**, layer-split. - It replaced the vllm-radiance MXFP4 replica pair + routing proxy ([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in the same day the pair went live. It is also the box's **only** LLM runtime: the @@ -102,7 +102,7 @@ bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc) | File | Bytes | Role | |---|---:|---| | `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB | -| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB | +| `Qwen3.8-27B-DSpark-Q8_0.gguf` | 1,359,284,737 | DSpark drafter, 1.27 GiB | Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified — a truncated 31 GB GGUF is otherwise a load-time crash). @@ -113,13 +113,13 @@ fine-tune: orcarouter's single refusal direction projected out of 131 tensors, vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image endpoint is frank-small. -**The file also carries the model's MTP head** (`blk.64.nextn.*`). With -`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the +**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under +`--spec-type draft-dspark` those tensors are deliberately **not** loaded — the log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` — -because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining) -`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the -head in; it is not enabled by default because the DFlash2 path is the measured -one. +because DSpark drafts from its own 5-layer sidecar. Switching (or chaining) +`--spec-type draft-mtp,draft-dspark` is a one-line env change that pulls the +head in; it is not enabled because the DSpark path is the measured one and +because a chain cannot hold two context-backed drafters at all (below). ## Deployment @@ -143,39 +143,107 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `40,25` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter (see the swap section) | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | -| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context | +| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context | +| `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance | +| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | -| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows | +| `--spec-type` / `--spec-draft-n-max` | `draft-dspark,ngram-mod` / `6` | DSpark blocks are 7 tokens, so 6 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer | | `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers | | `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint | | `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited | | `--no-mmproj` | — | text only | +## The DSpark swap (2026-09-16, hours after the cutover) + +The endpoint went live on **z-lab DFlash2** and was switched to +**magnitudedev/Qwen3.8-27B-DSpark** the same day. Both are 5-layer +block-diffusion sidecars, but they differ in ways that decide the deployment: + +| | DFlash2 (before) | DSpark (deployed) | +|---|---|---| +| file | 2,056,414,816 B — 1.92 GiB | 1,359,284,737 B — 1.27 GiB | +| draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) | +| attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector | +| draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context | + +**The draft context is sized from the target's.** `cparams.n_ctx = +llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it +independently, so a drafter that declares no sliding window caches the entire +target window on whichever device `--spec-draft-device` names. DSpark's KV is +5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position. + +That cost has to be paid somewhere, and the failure was instructive: at a 1:1 +split the model **loaded fine, answered `/health`, and then died on the first +draft decode** with `ROCm error: out of memory` inside +`common_speculative_impl_draft_dflash::process` — ROCm1 sat at 33.9 GB of +34.2 GB. A unit that crash-loops with an allocation failure *after* a +successful load is a sizing failure, not a broken drafter. + +Read `sliding_window` out of a drafter's GGUF header before sizing a +spec-decode deployment — 40 lines of `struct` parsing is enough, and the +flag-level configuration looks identical either way. + +### The split then had to move again for the batch + +Tripling the batch (2048/512 -> 6144/1536) made the two cards diverge: at +262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against ROCm0's +~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs — which +left the draft-carrying card at **0.08 GiB free** against ROCm0's 3.78 under +the 38:27 split that had been balanced at the smaller batch. One `-ts` step is +worth ~1.5 GiB at this size, so the balance was found by measurement, not +arithmetic: + +| `-ts` | GPU 0 free | GPU 1 free | +|---|---:|---:| +| 38,27 | 3.78 GiB | 0.08 GiB | +| 41,24 | 1.47 GiB | 2.38 GiB | +| **40,25 — deployed** | **1.88 GiB** | **1.88 GiB** | + +The balance is also what **maximises the context**: an unbalanced split has to +buy its headroom out of `CTX_SIZE`, and `CTX_SIZE` is already at the model's +trained window. The operator's floor for this configuration is 1.5 GiB per +card, and `scripts/frank-medium.env`'s sizing table carries the measurements +that 8192/2048 does *not* fit here at all (its 2020 MiB pp buffer fails to +allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB. + +> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its +> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the +> engine surfaced `ROCm error: an illegal memory access was encountered` -> +> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer +> split's hot path. It has not reproduced (38:27 served the whole e2e gate and +> a burst of requests; 40:25 has served the verification battery and a deploy +> gate), and it is **not** an OOM: no amount of headroom would prevent it. + ## Verified serving behaviour (2026-09-16) -- Load: `model loaded` ~29 s after start with a warm page cache, then +- Load: `model loaded` ~25 s after start with a warm page cache, then `listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s. -- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports - `modalities: {vision: false}`, one slot, and the sampler defaults above. -- DFlash2 engaged — the init line is the proof, not the flag: +- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports + `modalities: {vision: false}` and the sampler defaults above. +- DSpark engaged — the init lines are the proof, not the flags: ``` - common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash' - common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00 - common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true + common_speculative_init_result: loading draft model '.../Qwen3.8-27B-DSpark-Q8_0.gguf' + common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark' + common_speculative_impl_draft_dflash: - n_max=6, n_min=0, p_min=0.00 + common_speculative_impl_draft_dflash: - block_size=7, mask_token_id=248077, n_extract=5, sample_from_anchor=true ``` - `block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is - what distinguishes it from DFlash 1. -- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on - GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention - layers' KV and the compute buffers all sit on the second card, which keeps - ~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB. -- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one - slot: the raised window is real, not just declared, and it is 22 % beyond the - 114688 the vLLM predecessor could offer. + `block_size=7` and `mask_token_id=248077` are DSpark's own metadata. The + implementation still logs under the `draft_dflash` symbol because dspark *is* + that code path plus the Markov head — the loader prefers a dspark sidecar + when one is present. +- Both cards hold the model and they are **balanced**: at 262144 with the + deployed batch, `rocm-smi` reports **29.98 GiB used on each card** of + 31.86 GiB — 1.88 GiB free on each, measured with two concurrent 1832-token + prompts in flight. The 38:27 split that preceded the batch change was + lopsided (3.78 / 0.08 GiB free); see the swap section above. +- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured + through a *single* slot, at the previous configuration: the raised window is + real rather than declared, and it is 22 % beyond the 114688 the vLLM + predecessor could offer. The window is now split over two slots. - **Nothing is replicated.** The loader does not print a per-device ledger at this verbosity, so this is measured instead of read: raising CTX_SIZE from @@ -213,9 +281,12 @@ no proxy. ### What the drafter is worth — and why MTP is not the alternative -Measured 2026-09-16 with `scripts/bench-spec-ab.py`, four configurations of the -same binary, identical prompts at `temperature 0`, unit stopped between runs -(one host, one copy of a 29 GiB model). 1k-token prompt, 512-token budget: +Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter: +four configurations of the same binary, identical prompts at `temperature 0`, +unit stopped between runs (one host, one copy of a 29 GiB model). 1k-token +prompt, 512-token budget. These are the numbers that selected a trained +multi-token drafter over the model's own MTP head; DSpark replaced DFlash2 for +the reasons in the swap section, and it is that conclusion which carried over. | Configuration | Decode | Drafting | VRAM (card 1) | |---|---:|---|---:| @@ -244,14 +315,19 @@ same binary, identical prompts at `temperature 0`, unit stopped between runs ### Chaining the draftless n-gram matcher (deployed) -`ngram-mod` **can** ride beside DFlash2, and it is enabled. The reason is in -`common/speculative.cpp`: the n-gram types are added to the config vector +`ngram-mod` **can** ride beside a context-backed drafter, and it is enabled. +The reason is in `common/speculative.cpp`: the n-gram types are added to the config vector unconditionally while every model-backed drafter is gated on `ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with the drafter over one. (MTP is the opposite: it *is* context-backed, which is why `draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types first, then the drafters — also means CLI order is irrelevant. +The measurements below are DFlash2's; the chain is now `draft-dspark,ngram-mod`. +The matcher's behaviour — zero drafts on prose, 100 % on echo — is a property +of the matcher, not of the drafter beside it, but re-run +`scripts/bench-spec-ab.py` if the DSpark numbers themselves matter. + Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a proposer can and cannot match: @@ -307,14 +383,14 @@ agent loops re-sending a ~30k-token document concurrently: | family 1 | 24,799 ms | 30,697 ms | | family 0 | 51,424 ms | 8,015 ms | -Read that against the prefill table, not as a latency regression: one slot -means the second family's first turn waits for the first (25 s of cold 30k -prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that -the cache could only partly serve. **The single slot is now the limiting -resource for concurrent agent loops**, which is the trade this deployment makes -for giving one conversation the whole 262144-token window instead of splitting -two. Raising `--parallel` splits the window (and the KV) between slots; it does -not add throughput. +Read that against the prefill table, not as a latency regression: with **one** +slot the second family's first turn waited for the first (25 s of cold 30k +prefill + the wait ≈ 51 s), and the last number — 8.0 s — was a re-prefill the +cache could only partly serve. That was the 1-slot configuration. The +deployment now runs **two slots of 131072**, so a second agent loop has a slot +of its own instead of queueing, at the cost of halving the window each one +gets. `--parallel` splits the context and the KV between slots; it does not add +throughput, so the trade is concurrency against conversation length. ## Observability @@ -348,11 +424,12 @@ gets the server defaults above; a client that sends its own wins. Text only. - **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be `0,1`; the iGPU enumerates as device 2 and must stay hidden. - **Drafting not engaged** — read the journal for - `common_speculative_impl_draft_dflash`; a missing drafter file makes the - serve script exit before llama-server starts, and a non-DFlash2 drafter (or - an older llama.cpp) fails the load instead. + `common_speculative_impl_draft_dflash` (DFlash2 and DSpark both log under that + symbol); a missing drafter file makes the serve script exit before + llama-server starts, and a drafter whose GGUF the build does not recognise + fails the load instead. - **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is - correct for `draft-dflash`. It is not a truncated download: + correct for `draft-dspark`. It is not a truncated download: `bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes. - **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a gfx1200 sibling binary links, loads the model and then segfaults on these