frank-medium: pay for DSpark's draft KV with a 38:27 split
DSpark declares no sliding window, so its draft context -- built with n_ctx = llama_n_ctx(ctx_tgt) -- caches the whole target window: 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB/position, i.e. 5.0 GiB at 262144, where DFlash2's sliding_window 2048 cost ~40 MiB. At a 1:1 split that put ROCm1 at 33.9 GB of 34.2 GB and the first draft graph capture died inside common_speculative_impl_draft_dflash::process with "ROCm error: out of memory" -- after the model had loaded and /health had answered, which is why the deploy looked like a crash-loop rather than a sizing failure. Handing three target layers to ROCm0 restores headroom on both cards at the full window. Measured at 262144: 28.52 GB / 30.71 GB, i.e. 5.30 GiB and 3.26 GiB free (was 2.7 GB free on the draft-carrying card with 1:1 + DFlash2). The unit seam now models the busier card as a skewed share plus the drafter's full-context KV, calibrated to the rocm-smi figure. AGENTS.md gains the durable lesson: read sliding_window out of the drafter's GGUF header before sizing a spec-decode deployment.
This commit is contained in:
1 parent
46b56ab5e8
commit
b6d4914fcc
3 files changed
+64
-25
No files matched your search
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process, 262144 ctx / 1 slot, text only | 7770 |
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (38:27), 262144 ctx / 1 slot, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
|
||||
|
||||
@@ -298,14 +298,14 @@ len ≈ 3` (MTP engaged); an image request is answered from the projector while
|
||||
the halogen side refuses images with a named reason; 22,362 MiB of the card's
|
||||
24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
|
||||
|
||||
### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
|
||||
### frank-medium — llama.cpp layer split, DSpark + ngram-mod (Dankserver, ACTIVE)
|
||||
|
||||
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
|
||||
systemd unit `scripts/frank-medium.service`) serving
|
||||
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
|
||||
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless
|
||||
**ngram-mod** matcher, in **one process across both R9700s**, `--split-mode
|
||||
layer`, 262144 ctx / 1 slot, text only.
|
||||
(29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the
|
||||
draftless **ngram-mod** matcher, in **one process across both R9700s**,
|
||||
`--split-mode layer` at a **38:27 tensor split**, 262144 ctx / 1 slot, text only.
|
||||
|
||||
**Layer split is why both cards now serve ONE model.** Tensor parallelism is
|
||||
impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
|
||||
@@ -496,6 +496,19 @@ linked runbooks.
|
||||
doesn't contain MTP layers"*, because MTP builds its context against whatever
|
||||
model the draft context holds. Two context-backed types cannot share one
|
||||
process, and MTP's head only loads when MTP **is** the draft model.
|
||||
- **A drafter's KV scales with the TARGET's context, and its window policy
|
||||
decides how much that costs.** The draft context is built with
|
||||
`cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
|
||||
no flag to cap it — so a drafter that declares NO sliding window caches the
|
||||
whole window on whatever device `--spec-draft-device` names. Swapping DFlash2
|
||||
(`dflash.attention.sliding_window = 2048`, ~40 MiB of draft KV) for DSpark
|
||||
(no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position,
|
||||
so **5.0 GiB at 262144**) put ROCm1 at 33.9 GB of 34.2 GB and killed the
|
||||
FIRST draft graph capture with `ROCm error: out of memory` — after the model
|
||||
had loaded and `/health` had answered. Read `sliding_window` out of the
|
||||
drafter's GGUF header before sizing any spec-decode deployment, and prefer a
|
||||
skewed `-ts` over a context cut when the full window matters. See
|
||||
[[frank-medium-llamacpp-dflash2]].
|
||||
- **ngram-mod prints no startup banner.** The load log shows the drafter's
|
||||
`adding speculative implementation 'draft-dflash'` and nothing at all for the
|
||||
matcher, so a chain can look unengaged while it is working. Identify it
|
||||
@@ -587,7 +600,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
|
||||
| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
|
||||
| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| frank-medium (llama.cpp + DSpark, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
|
||||
Reference in new issue
Block a user