dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the frank-medium runbook catches up with DSpark, 2 slots and the rebalance

Browse Source 
The note still described DFlash2, one slot at 262144 and a 1:1 split -- all of
which the day's work replaced. It now carries:

- the DSpark swap with the table that matters (1.27 vs 1.92 GiB, block_size 7 vs
  8 so n_max 6 vs 7, a Markov head instead of the candidate selector, and NO
  sliding window) and the rule behind the sizing: the draft context is built
  with n_ctx = llama_n_ctx(ctx_tgt), so a drafter without a window caches the
  whole target context -- 5.0 GiB here, against DFlash2's ~40 MiB.
- the failure that rule produces when ignored: load succeeds, /health answers,
  and the first draft decode dies with an OOM at 1:1.
- why the split moved a second time: tripling the batch costs ROCm1 about twice
  what it costs ROCm0, with the 38,27 -> 41,24 -> 40,25 measurement table.
- the deployed flags, the 2-slot shape (n_slots = 2, n_ctx_slot = 131072), the
  DSpark init lines, and balanced 29.98 / 29.98 GiB cards.
- honest scoping on the older measurements: the throughput, drafter-worth and
  ngram tables were taken on DFlash2, and the agent-loop numbers on one slot.
- the 39:26 page-fault signature as a thing to watch, marked as NOT an OOM.
This commit is contained in:
Frank
2026-09-16 20:21:31 -07:00
parent b312042706
commit 718a283afc
1 changed files with 126 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+126 -49
View File
@@ -1,12 +1,12 @@
---
type: configuration
title: frank-medium llama.cpp + DFlash2 Runbook
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards.
title: frank-medium llama.cpp Runbook (DSpark drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards, 262144 ctx over two slots.
tags:
- frank-medium
- dankserver
- llama-cpp
- dflash2
- dspark
- qwen35
- gfx1201
- r9700
@@ -14,13 +14,13 @@ tags:
timestamp: 2026-09-16
---
# frank-medium — llama.cpp + DFlash2
# frank-medium — llama.cpp (DSpark drafter)
The ACTIVE frank-medium deployment since 2026-09-16: **mainline llama.cpp**
serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at `UD-Q8_K_XL`
with the **z-lab DFlash2** block-diffusion drafter, as **one process across
with the **magnitudedev DSpark** block-diffusion drafter (it replaced z-lab
DFlash2 the same day — see the swap section below), as **one process across
both Radeon AI PRO R9700 cards**, layer-split.
It replaced the vllm-radiance MXFP4 replica pair + routing proxy
([[frank-medium-radiance-mxfp4]], the record and the PCIe-atomics evidence) in
the same day the pair went live. It is also the box's **only** LLM runtime: the
@@ -102,7 +102,7 @@ bash scripts/build-llama-rdna4.sh # ~25 min on this box, JOBS=$(nproc)
| File | Bytes | Role |
|---|---:|---|
| `Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf` | 31,457,990,112 | target, 29.29 GiB |
| `Qwen3.8-27B-DFlash2-Q8_0.gguf` | 2,056,414,816 | DFlash2 drafter, 1.92 GiB |
| `Qwen3.8-27B-DSpark-Q8_0.gguf` | 1,359,284,737 | DSpark drafter, 1.27 GiB |
Fetched by `scripts/download-frank-medium-gguf.sh` (resumable, byte-verified —
a truncated 31 GB GGUF is otherwise a load-time crash).
@@ -113,13 +113,13 @@ fine-tune: orcarouter's single refusal direction projected out of 131 tensors,
vision tower untouched. **Text only here** (`--no-mmproj`); the fleet's image
endpoint is frank-small.
**The file also carries the model's MTP head** (`blk.64.nextn.*`). With
`--spec-type draft-dflash` those tensors are deliberately **not** loaded — the
**The file also carries the model's MTP head** (`blk.64.nextn.*`). Under
`--spec-type draft-dspark` those tensors are deliberately **not** loaded — the
log says `model has unused tensor blk.64.nextn.eh_proj.weight ... ignoring` —
because DFlash2 drafts from its own 5-layer drafter. Switching (or chaining)
`--spec-type draft-mtp,draft-dflash` is a one-line env change that pulls the
head in; it is not enabled by default because the DFlash2 path is the measured
one.
because DSpark drafts from its own 5-layer sidecar. Switching (or chaining)
`--spec-type draft-mtp,draft-dspark` is a one-line env change that pulls the
head in; it is not enabled because the DSpark path is the measured one and
because a chain cannot hold two context-backed drafters at all (below).
## Deployment
@@ -143,39 +143,107 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `1,1` | both cards, one model, no collective |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `40,25` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter (see the swap section) |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `262144` / `1` | one slot carries the model's whole configured window; `--parallel` **divides** the context |
| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context |
| `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling llama.cpp allows |
| `--spec-type` / `--spec-draft-n-max` | `draft-dspark,ngram-mod` / `6` | DSpark blocks are 7 tokens, so 6 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
| `--spec-draft-device` | `ROCm1` | the drafter rides the card that ends up holding the target's last layers |
| `--temp` / `--top-p` / `--top-k` | `0.7` / `0.95` / `20` | the values the vLLM predecessor served on this endpoint |
| `--reasoning-format` / `--reasoning-budget` | `deepseek` / `-1` | trace into `reasoning_content`, unlimited |
| `--no-mmproj` | — | text only |
## The DSpark swap (2026-09-16, hours after the cutover)
The endpoint went live on **z-lab DFlash2** and was switched to
**magnitudedev/Qwen3.8-27B-DSpark** the same day. Both are 5-layer
block-diffusion sidecars, but they differ in ways that decide the deployment:
| | DFlash2 (before) | DSpark (deployed) |
|---|---|---|
| file | 2,056,414,816 B — 1.92 GiB | 1,359,284,737 B — 1.27 GiB |
| draft depth | `block_size 8` -> `n_max 7` | `block_size 7` -> `n_max 6` (llama.cpp asserts `n_max <= block_size - 1`) |
| attention | `sliding_window 2048`, `causal false`, candidate selector (rank 256, top-k 16) | **no sliding window**; a Markov head in place of the selector |
| draft KV at 262144 | ~40 MiB — the window bounds it | **5.0 GiB** — the whole context |
**The draft context is sized from the target's.** `cparams.n_ctx =
llama_n_ctx(ctx_tgt)` in `common/speculative.cpp`, with no flag to cap it
independently, so a drafter that declares no sliding window caches the entire
target window on whichever device `--spec-draft-device` names. DSpark's KV is
5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position.
That cost has to be paid somewhere, and the failure was instructive: at a 1:1
split the model **loaded fine, answered `/health`, and then died on the first
draft decode** with `ROCm error: out of memory` inside
`common_speculative_impl_draft_dflash::process` — ROCm1 sat at 33.9 GB of
34.2 GB. A unit that crash-loops with an allocation failure *after* a
successful load is a sizing failure, not a broken drafter.
Read `sliding_window` out of a drafter's GGUF header before sizing a
spec-decode deployment — 40 lines of `struct` parsing is enough, and the
flag-level configuration looks identical either way.
### The split then had to move again for the batch
Tripling the batch (2048/512 -> 6144/1536) made the two cards diverge: at
262144 the bigger `--ubatch-size` cost **ROCm1 ~3.3 GiB against ROCm0's
~1.7 GiB** — ROCm1 holds the output layers *and* the drafter's graphs — which
left the draft-carrying card at **0.08 GiB free** against ROCm0's 3.78 under
the 38:27 split that had been balanced at the smaller batch. One `-ts` step is
worth ~1.5 GiB at this size, so the balance was found by measurement, not
arithmetic:
| `-ts` | GPU 0 free | GPU 1 free |
|---|---:|---:|
| 38,27 | 3.78 GiB | 0.08 GiB |
| 41,24 | 1.47 GiB | 2.38 GiB |
| **40,25 — deployed** | **1.88 GiB** | **1.88 GiB** |
The balance is also what **maximises the context**: an unbalanced split has to
buy its headroom out of `CTX_SIZE`, and `CTX_SIZE` is already at the model's
trained window. The operator's floor for this configuration is 1.5 GiB per
card, and `scripts/frank-medium.env`'s sizing table carries the measurements
that 8192/2048 does *not* fit here at all (its 2020 MiB pp buffer fails to
allocate on ROCm1) while 4096/1024 fits but leaves 1.11 GiB.
> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its
> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the
> engine surfaced `ROCm error: an illegal memory access was encountered` ->
> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer
> split's hot path. It has not reproduced (38:27 served the whole e2e gate and
> a burst of requests; 40:25 has served the verification battery and a deploy
> gate), and it is **not** an OOM: no amount of headroom would prevent it.
## Verified serving behaviour (2026-09-16)
- Load: `model loaded` ~29 s after start with a warm page cache, then
- Load: `model loaded` ~25 s after start with a warm page cache, then
`listening on http://0.0.0.0:7770`. The drafter adds ~2.5 s.
- Slot shape: `n_slots = 1, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}`, one slot, and the sampler defaults above.
- DFlash2 engaged — the init line is the proof, not the flag:
- Slot shape: `n_slots = 2, n_ctx_slot = 131072`; `/props` reports
`modalities: {vision: false}` and the sampler defaults above.
- DSpark engaged — the init lines are the proof, not the flags:
```
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dflash'
common_speculative_impl_draft_dflash: - n_max=7, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=8, mask_token_id=248070, n_extract=5, sample_from_anchor=true
common_speculative_init_result: loading draft model '.../Qwen3.8-27B-DSpark-Q8_0.gguf'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
common_speculative_impl_draft_dflash: - n_max=6, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=7, mask_token_id=248077, n_extract=5, sample_from_anchor=true
```
`block_size=8` and `n_extract=5` are DFlash2's own metadata; the selector is
what distinguishes it from DFlash 1.
- Both cards hold the model: at 262144 context `rocm-smi` reports **24.72 GB on
GPU 0 and 29.14 GB on GPU 1** of 34.21 GB each — the drafter, the attention
layers' KV and the compute buffers all sit on the second card, which keeps
~5.1 GB of headroom. At 131072 the same pair reads 19.89 / 24.30 GB.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) through the one
slot: the raised window is real, not just declared, and it is 22 % beyond the
114688 the vLLM predecessor could offer.
`block_size=7` and `mask_token_id=248077` are DSpark's own metadata. The
implementation still logs under the `draft_dflash` symbol because dspark *is*
that code path plus the Markov head — the loader prefers a dspark sidecar
when one is present.
- Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch, `rocm-smi` reports **29.98 GiB used on each card** of
31.86 GiB — 1.88 GiB free on each, measured with two concurrent 1832-token
prompts in flight. The 38:27 split that preceded the batch change was
lopsided (3.78 / 0.08 GiB free); see the swap section above.
- **140,069 prompt tokens prefilled in 134 s** (1043 tok/s) was measured
through a *single* slot, at the previous configuration: the raised window is
real rather than declared, and it is 22 % beyond the 114688 the vLLM
predecessor could offer. The window is now split over two slots.
- **Nothing is replicated.** The loader does not print a per-device ledger at
this verbosity, so this is measured instead of read: raising CTX_SIZE from
@@ -213,9 +281,12 @@ no proxy.
### What the drafter is worth — and why MTP is not the alternative
Measured 2026-09-16 with `scripts/bench-spec-ab.py`, four configurations of the
same binary, identical prompts at `temperature 0`, unit stopped between runs
(one host, one copy of a 29 GiB model). 1k-token prompt, 512-token budget:
Measured 2026-09-16 with `scripts/bench-spec-ab.py` on the **DFlash2** drafter:
four configurations of the same binary, identical prompts at `temperature 0`,
unit stopped between runs (one host, one copy of a 29 GiB model). 1k-token
prompt, 512-token budget. These are the numbers that selected a trained
multi-token drafter over the model's own MTP head; DSpark replaced DFlash2 for
the reasons in the swap section, and it is that conclusion which carried over.
| Configuration | Decode | Drafting | VRAM (card 1) |
|---|---:|---|---:|
@@ -244,14 +315,19 @@ same binary, identical prompts at `temperature 0`, unit stopped between runs
### Chaining the draftless n-gram matcher (deployed)
`ngram-mod` **can** ride beside DFlash2, and it is enabled. The reason is in
`common/speculative.cpp`: the n-gram types are added to the config vector
`ngram-mod` **can** ride beside a context-backed drafter, and it is enabled.
The reason is in `common/speculative.cpp`: the n-gram types are added to the config vector
unconditionally while every model-backed drafter is gated on
`ctx_dft != nullptr` — so n-gram needs no draft context and cannot collide with
the drafter over one. (MTP is the opposite: it *is* context-backed, which is why
`draft-mtp,draft-dflash` cannot load.) The vector's fixed order — ngram types
first, then the drafters — also means CLI order is irrelevant.
The measurements below are DFlash2's; the chain is now `draft-dspark,ngram-mod`.
The matcher's behaviour — zero drafts on prose, 100 % on echo — is a property
of the matcher, not of the drafter beside it, but re-run
`scripts/bench-spec-ab.py` if the DSpark numbers themselves matter.
Measured with `scripts/bench-spec-ab.py`, decode tok/s, three shapes that a
proposer can and cannot match:
@@ -307,14 +383,14 @@ agent loops re-sending a ~30k-token document concurrently:
| family 1 | 24,799 ms | 30,697 ms |
| family 0 | 51,424 ms | 8,015 ms |
Read that against the prefill table, not as a latency regression: one slot
means the second family's first turn waits for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — is a re-prefill that
the cache could only partly serve. **The single slot is now the limiting
resource for concurrent agent loops**, which is the trade this deployment makes
for giving one conversation the whole 262144-token window instead of splitting
two. Raising `--parallel` splits the window (and the KV) between slots; it does
not add throughput.
Read that against the prefill table, not as a latency regression: with **one**
slot the second family's first turn waited for the first (25 s of cold 30k
prefill + the wait ≈ 51 s), and the last number — 8.0 s — was a re-prefill the
cache could only partly serve. That was the 1-slot configuration. The
deployment now runs **two slots of 131072**, so a second agent loop has a slot
of its own instead of queueing, at the cost of halving the window each one
gets. `--parallel` splits the context and the KV between slots; it does not add
throughput, so the trade is concurrency against conversation length.
## Observability
@@ -348,11 +424,12 @@ gets the server defaults above; a client that sends its own wins. Text only.
- **`hipErrorNoDevice` / a card missing** — `HIP_VISIBLE_DEVICES` must be
`0,1`; the iGPU enumerates as device 2 and must stay hidden.
- **Drafting not engaged** — read the journal for
`common_speculative_impl_draft_dflash`; a missing drafter file makes the
serve script exit before llama-server starts, and a non-DFlash2 drafter (or
an older llama.cpp) fails the load instead.
`common_speculative_impl_draft_dflash` (DFlash2 and DSpark both log under that
symbol); a missing drafter file makes the serve script exit before
llama-server starts, and a drafter whose GGUF the build does not recognise
fails the load instead.
- **Why is `blk.64.*` "unused"?** That is the model's own MTP head and it is
correct for `draft-dflash`. It is not a truncated download:
correct for `draft-dspark`. It is not a truncated download:
`bash scripts/download-frank-medium-gguf.sh --verify-only` proves the bytes.
- **A rebuild segfaults at load** — the build is gfx1201-only on purpose; a
gfx1200 sibling binary links, loads the model and then segfaults on these