dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: back to the DFlash2 drafter, and rebalance to 36:29

Browse Source 
DSpark came off after a day. It was the better-looking file (1.27 GiB against
1.92) but it declares no sliding window, and the draft context is built with
n_ctx = llama_n_ctx(ctx_tgt) -- so its KV spanned the whole 262144-token window
(5.0 GiB) and cost ROCm1 four layers of split to carry. DFlash2's window bounds
its draft KV at 2048 positions (~40 MiB), which frees ~4.3 GiB on the
draft-carrying card.

The balance point moved with it, so it was re-measured at the deployed batch
(6144 x 1536, 2 slots, 262144) rather than interpolated:

  39,26  2.42 / 7.20     37,28  4.28 / 5.34     35,30  5.14 / 4.48
  38,27  3.85 / 5.77     36,29  4.71 / 4.91     34,31  6.57 / 3.05

36:29 is the minimum (0.20 GiB apart) and leaves 3.2 GiB above the operator's
1.5 GiB floor on each card, so the full window stays. One -ts step is one layer,
and qwen35's layers are not the same size -- 0.43 GiB on a DeltaNet boundary,
1.43 GiB on a full-attention one -- which is why the table exists.

Also in here:
- the fetch script pointed at the wrong file (DSpark) AND declared the wrong
  size for it: 1359284737 against the Hub's 1455376576, 96 MB short, which made
  --verify-only fail on a payload that was in fact correct. Repointed at z-lab
  DFlash2 Q8_0, 2056414816, and both files' sha256 were checked against the
  Hub's LFS oids (host == Hub for both).
- the seam's fit was recalibrated for a window-bounded draft KV and for the new
  point; it now passes 36:29 and fails every other split measured, including
  1:1. Its ubatch term is an extrapolation and it cannot model contiguity, which
  the comment says out loud.
- the serve script's dead dspark-only tunables (n-min, p-min, p-split, poll)
  are gone; the build's defaults apply.
This commit is contained in:
Frank
2026-09-16 20:59:20 -07:00
parent 718a283afc
commit 91546dbb95
5 changed files with 129 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+30 -16
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + magnitudedev Qwen3.8-27B-DSpark Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (40:25), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
@@ -298,16 +298,19 @@ len ≈ 3` (MTP engaged); an image request is answered from the projector while
the halogen side refuses images with a named reason; 22,362 MiB of the card's
24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
### frank-medium — llama.cpp layer split, DSpark + ngram-mod (Dankserver, ACTIVE)
### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE)
Mainline llama.cpp at **`fb27a525`** (a gfx1201 HIP build, `Root=`-less root
systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
(29.29 GiB) with the **magnitudedev DSpark** `Q8_0` drafter chained to the
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
draftless **ngram-mod** matcher, in **one process across both R9700s**,
`--split-mode layer` at a **40:25 tensor split** (the balance point for the
`--split-mode layer` at a **36:29 tensor split** (the balance point for the
deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only.
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split.
**Layer split is why both cards now serve ONE model.** Tensor parallelism is
impossible (see the PCIe lesson below), and so is vLLM's pipeline parallelism —
@@ -509,17 +512,28 @@ linked runbooks.
FIRST draft graph capture with `ROCm error: out of memory` — after the model
had loaded and `/health` had answered. Read `sliding_window` out of the
drafter's GGUF header before sizing any spec-decode deployment, and prefer a
skewed `-ts` over a context cut when the full window matters. See
[[frank-medium-llamacpp-dflash2]].
- **The compute buffers of a bigger `--ubatch-size` land ~4x heavier on the card
that holds the output layers and the drafter.** On frank-medium (2026-09-16)
ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, which left the
draft-carrying card at 0.08 GiB free under a split that was balanced at the
smaller batch. A split balanced at one batch is unbalanced at another: ~1.5
GiB of load moves per `-ts` step at this size, so re-measure the balance after
changing the batch (and remember 8192/2048 does not fit at 262144 at all --
its 2020 MiB pp buffer fails to allocate). See
skewed `-ts` over a context cut when the full window matters. The pair was
swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on
the card that holds the output layers and the drafter.** On frank-medium
(2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7,
which left the draft-carrying card at 0.08 GiB free under a split that was
balanced at the smaller batch. A split balanced at one batch is unbalanced at
another, so re-measure the balance after changing the batch. (8192/2048 failed
to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1
had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and
it has not been re-measured since.)
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
(0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both
measured on frank-medium. Balance by measurement, never by interpolating
between two splits; an affine fit reproduces the deployed point and overstates
the gaps in between. `tests/frank_medium_unit.bats` encodes that fit as a
gate. See [[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
`adding speculative implementation 'draft-dflash'` and nothing at all for the
matcher, so a chain can look unengaged while it is working. Identify it
@@ -628,7 +642,7 @@ Active:
|---------|---------|
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
| frank-medium (llama.cpp + DSpark, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |