dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the runbook records the DSpark excursion as an excursion

Browse Source 
The note described DSpark as the deployed drafter and 40:25 as the split; both
were true for a few hours. It now leads with DFlash2 at 36:29 and keeps the
excursion as the record it is:

- the drafter comparison table with DSpark's REAL size (1,455,376,576, not the
  1,359,284,737 that went into the fetch script), and why it lost: no sliding
  window, so its draft KV spanned the whole context.
- the split's two moves as one story: the batch pushed it to 40:25, the drafter
  swap pulled it back to 36:29, with both measurement tables.
- the layer-size law (0.43 GiB DeltaNet, 1.43 GiB attention) as the reason the
  balance is measured and not interpolated.
- the verified-behaviour section now carries the LIVE numbers: 4.71 / 4.91 GiB
  free (4.43 / 4.61 under load, 0.18 GiB gap), the DFlash2 init lines, and
  draft acceptance from both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0).
- the fault note scopes its own evidence: 39:26 was loaded, not exercised.
This commit is contained in:
Frank
2026-09-16 21:01:45 -07:00
parent 91546dbb95
commit f917a2ef55
1 changed files with 97 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+97 -62
View File
@@ -1,11 +1,12 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (DSpark drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards, 262144 ctx over two slots.
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, 262144 ctx over two slots.
tags:
- frank-medium
- dankserver
- llama-cpp
- dflash2
- dspark
- qwen35
- gfx1201
@@ -14,13 +15,13 @@ tags:
timestamp: 2026-09-16
---
# frank-medium — llama.cpp (DSpark drafter)
# frank-medium — llama.cpp (DFlash2 drafter)
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **magnitudedev DSpark** block-diffusion drafter (it replaced z-lab
DFlash2 the same day — see the swap section below), as **one process across
both Radeon AI PRO R9700 cards**, layer-split.
with the **z-lab DFlash2** block-diffusion drafter, as **one process across both
Radeon AI PRO R9700 cards**, layer-split. DSpark was tried in the drafter's
place for a few hours the same day and reverted — see *The DSpark excursion*.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
@@ -102,7 +103,7 @@ bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DSpark-Q8_0.gguf` | 1,359,284,737 | DSpark drafter, 1.27 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).
@@ -114,12 +115,12 @@ vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.
**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
`--spec-type draft-dspark` those tensors are deliberately **not** loaded — the
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DSpark drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dspark` is a one-line env change that pulls the
head in; it is not enabled because the DSpark path is the measured one and
because a chain cannot hold two context-backed drafters at all (below).
because DFlash2 drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled because the DFlash2 path is the measured one (below)
and because a chain cannot hold two context-backed drafters at all.
## Deployment
@@ -143,31 +144,38 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `40,25` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter (see the swap section) |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context |
| `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dspark,ngram-mod` / `6` | DSpark blocks are 7 tokens, so 6 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--no-mmproj` | — | text only |
## The DSpark swap (2026-09-16, hours after the cutover)
## The DSpark excursion (2026-09-16, tried and reverted)
The endpoint went live on **z-lab DFlash2** and was switched to
**magnitudedev/Qwen3.8-27B-DSpark** the same day. Both are 5-layer
block-diffusion sidecars, but they differ in ways that decide the deployment:
The endpoint went live on **z-lab DFlash2**, was switched to
**magnitudedev/Qwen3.8-27B-DSpark** for a few hours the same day, and was put
back. Both are 5-layer block-diffusion sidecars, and their differences are what
decided it:
| | DFlash2 (before) | DSpark (deployed) |
| | DFlash2 (deployed) | DSpark (tried, reverted) |
|---|---|---|
| file | 2,056,414,816 B — 1.92 GiB | 1,359,284,737 B — 1.27 GiB |
| file | 2,056,414,816 B — 1.92 GiB | 1,455,376,576 B — 1.36 GiB |
| draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) |
| attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector |
| draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context |
> **A fetch-script bug this threw up.** The DSpark file's size was recorded as
> 1,359,284,737 when it was fetched — 96 MB short of the Hub's 1,455,376,576 —
> and that wrong number became the fetch script's declared size, so
> `--verify-only` failed on a payload that was in fact byte-correct. Both
> drafters' sha256 have since been checked against the Hub's LFS oids and match.
**The draft context is sized from the target's.** `cparams.n_ctx =
llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it
independently, so a drafter that declares no sliding window caches the entire
@@ -185,36 +193,59 @@ Read `sliding_window` out of a drafter's GGUF header before sizing a
spec-decode deployment — 40 lines of `struct` parsing is enough, and the
flag-level configuration looks identical either way.
### The split then had to move again for the batch
### The split moved twice: once for the batch, once for the drafter
Tripling the batch (2048/512 -> 6144/1536) made the two cards diverge: at
262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against ROCm0's
~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs — which
left the draft-carrying card at **0.08 GiB free** against ROCm0's 3.78 under
the 38:27 split that had been balanced at the smaller batch. One `-ts` step is
worth ~1.5 GiB at this size, so the balance was found by measurement, not
arithmetic:
**Under DSpark.** Tripling the batch (2048/512 -> 6144/1536) made the two cards
diverge: at 262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against
ROCm0's ~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs —
which left the draft-carrying card at 0.08 GiB free against ROCm0's 3.78 under
the 38:27 split that had been balanced at the smaller batch:
| `-ts` | GPU 0 free | GPU 1 free |
|---|---:|---:|
| 38,27 | 3.78 GiB | 0.08 GiB |
| 41,24 | 1.47 GiB | 2.38 GiB |
| **40,25 — deployed** | **1.88 GiB** | **1.88 GiB** |
| 40,25 | 1.88 GiB | 1.88 GiB |
The balance is also what **maximises the context**: an unbalanced split has to
buy its headroom out of `CTX_SIZE`, and `CTX_SIZE` is already at the model's
trained window. The operator's floor for this configuration is 1.5 GiB per
card, and `scripts/frank-medium.env`'s sizing table carries the measurements
that 8192/2048 does *not* fit here at all (its 2020 MiB pp buffer fails to
allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB.
**Then DSpark came off**, which frees ~4.3 GiB on ROCm1 (5.0 GiB of draft KV
out, 0.65 GiB of heavier DFlash2 weights in) and moves the balance the other
way. Re-measured at the same batch, the same two slots and the same window:
| `-ts` | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|
| 39,26 | 2.42 GiB | 7.20 GiB | 4.78 |
| 38,27 | 3.85 GiB | 5.77 GiB | 1.92 |
| 37,28 | 4.28 GiB | 5.34 GiB | 1.06 |
| **36,29 — deployed** | **4.71 GiB** | **4.91 GiB** | **0.20** |
| 35,30 | 5.14 GiB | 4.48 GiB | 0.66 |
| 34,31 | 6.57 GiB | 3.05 GiB | 3.52 |
**One `-ts` step is one layer, and this stack's layers are not the same size.**
qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth
**~0.43 GiB** on a DeltaNet boundary and **~1.43 GiB** on a full-attention one
(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144). Both fell out
of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an
attention layer. Balance by measurement, never by interpolating.
36:29 is both the minimum and the point that leaves the most room: it clears the
operator's 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought
out of `CTX_SIZE` and the model's full 262144-token window stays.
A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not
be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure,
not a total-bytes one, and it has **not** been re-measured with DFlash2's ~4.9
GiB of headroom. (`tests/frank_medium_unit.bats` fits the deployed point and
predicts it still would not fit, but that is an extrapolation of a coefficient
fitted at ubatch 1536.)
> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its
> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the
> engine surfaced `ROCm error: an illegal memory access was encountered` ->
> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer
> split's hot path. It has not reproduced (38:27 served the whole e2e gate and
> a burst of requests; 40:25 has served the verification battery and a deploy
> gate), and it is **not** an OOM: no amount of headroom would prevent it.
> split's hot path. It has not reproduced (36:29 has since served the whole
> verification battery and the e2e gate; 39:26 itself was only *loaded* at the
> deployed batch, not exercised), and it is **not** an OOM: no amount of
> headroom would prevent it.
## Verified serving behaviour (2026-09-16)
@@ -222,24 +253,30 @@ allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB.
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}` and the sampler defaults above.
- DSpark engaged — the init lines are the proof, not the flags:
- DFlash2 engaged — the init lines are the proof, not the flags:
```
common_speculative_init_result: loading draft model '.../Qwen3.8-27B-DSpark-Q8_0.gguf'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
common_speculative_impl_draft_dflash: - n_max=6, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=7, mask_token_id=248077, n_extract=5, sample_from_anchor=true
common_speculative_init_result: loading draft model '/home/dank/models/Qwen3.8-27B-DFlash2-GGUF/Qwen3.8-27B-DFlash2-Q8_0.gguf'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
```
`block_size=7` and `mask_token_id=248077` are DSpark's own metadata. The
implementation still logs under the `draft_dflash` symbol because dspark *is*
that code path plus the Markov head — the loader prefers a dspark sidecar
when one is present.
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the candidate
selector is what distinguishes it from DFlash 1. DSpark logs under the same
`draft_dflash` symbol with `block_size=7` and a Markov head, so read the
numbers, not the implementation name.
- Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch, `rocm-smi` reports **29.98 GiB used on each card** of
31.86 GiB — 1.88 GiB free on each, measured with two concurrent 1832-token
prompts in flight. The 38:27 split that preceded the batch change was
lopsided (3.78 / 0.08 GiB free); see the swap section above.
deployed batch and split, `rocm-smi` reports **4.71 GiB free on GPU 0 and
4.91 GiB on GPU 1** of 31.86 GiB each — the same figures the balance search
predicted to 0.01 GiB. Under load (two concurrent requests, one per slot) they
sit at 4.43 / 4.61, a 0.18 GiB gap, and both are far above the operator's
1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
`0.38776 (57 / 147), mean len = 3.71` on slot 0, from one concurrent pair.
Acceptance moves with the text; the point is that neither slot is idle.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured
through a *single* slot, at the previous configuration: the raised window is
real rather than declared, and it is 22 % beyond the 114688 the vLLM
@@ -281,12 +318,11 @@ no proxy.
### What the drafter is worth — and why MTP is not the alternative
Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter:
four configurations of the same binary, identical prompts at `temperature 0`,
unit stopped between runs (one host, one copy of a 29 GiB model). 1k-token
prompt, 512-token budget. These are the numbers that selected a trained
multi-token drafter over the model's own MTP head; DSpark replaced DFlash2 for
the reasons in the swap section, and it is that conclusion which carried over.
Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter —
the one deployed now: four configurations of the same binary, identical prompts
at `temperature 0`, unit stopped between runs (one host, one copy of a 29 GiB
model). 1k-token prompt, 512-token budget. These are the numbers that selected a
trained multi-token drafter over the model's own MTP head.
| Configuration | Decode | Drafting | VRAM (card 1) |
|---|---:|---|---:|
@@ -323,10 +359,9 @@ the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.
The measurements below are DFlash2's; the chain is now `draft-dspark,ngram-mod`.
The matcher's behaviour — zero drafts on prose, 100 % on echo — is a property
of the matcher, not of the drafter beside it, but re-run
`scripts/bench-spec-ab.py` if the DSpark numbers themselves matter.
The measurements below are DFlash2's — the deployed configuration. (During the
DSpark excursion the chain was `draft-dspark,ngram-mod`; the matcher's behaviour
is a property of the matcher, not of the drafter beside it.)
Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:
@@ -429,7 +464,7 @@ gets the server defaults above; a client that sends its own wins. Text only.
llama-server starts, and a drafter whose GGUF the build does not recognise
fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
correct for `draft-dspark`. It is not a truncated download:
correct for `draft-dflash`. It is not a truncated download:
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
gfx1200 sibling binary links, loads the model and then segfaults on these