From 6569c63ec4cead915143bf43397a7425938b2879 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 18:58:06 -0700 Subject: [PATCH] frank-medium: shift the layer split to 35:30, leaning work onto the x16 card The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through bridges negotiating Gen4 x1 (~1.97 GB/s). With the MTP drafter, output layers and heavier compute buffers pinned to the last-layers card, live rocm-smi read 98% busy on the x1 card vs 40% on the x16 card. Moving one DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card; the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack. 36:29 fails both gates, so this lever is spent. --- AGENTS.md | 13 +++++-- scripts/frank-medium.env | 30 ++++++++++++--- .../LLM/frank-medium-llamacpp-dflash2.md | 38 +++++++++++++++++-- tests/frank_medium_unit.bats | 27 +++++++------ 4 files changed, 83 insertions(+), 25 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f0a1b86..4369c19 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 | +|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770| | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 | @@ -335,8 +335,15 @@ systemd unit `scripts/frank-medium.service`) serving (29.29 GiB) with the model's OWN **MTP head** (`blk..nextn.*` in the file; the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one `SPEC_TYPE` change brings the faster chain back), in **one process across both -R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point -at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision +R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved +to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric: +ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1 +(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the +drafter, output layers and heavier compute buffers pinned to the +last-layers card, the bottlenecked card was running 98% while the x16 card +sat at 40%. 36:29 would cross a full-attention layer and fails both the +1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this +pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision via the BF16 mmproj. DSpark was put up in the drafter's place the same day and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index 9da5828..6a38573 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -141,12 +141,30 @@ HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 SPLIT_MODE=layer -# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head -# and its KV ride ROCm1 with the last layers, so the split sends MORE layers -# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step -# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not -# move it without re-measuring. -TENSOR_SPLIT=34,31 +# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card. The +# two R9700s do NOT have symmetric PCIe paths. ROCm0 (03:00.0) hangs off +# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1 +# (0b:00.0) rides the B650 chipset through bridges that each negotiate +# Gen4 x1 (~1.97 GB/s). Live rocm-smi showed the x1 card at 98% busy +# against 40% on the x16 card: the MTP drafter, the output layers and the +# heavier compute buffers are structurally pinned to the card holding the +# LAST layers (the draft context shares the target's weights there), so +# the bottlenecked card was doing most of the work. Moving one layer to +# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet -- +# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy +# card the bigger share. The layer-split boundary itself is not the cost +# (~5.2 MB of activation per 512-token ubatch rides the wire), but the +# card behind the x1 link should not be the saturated one while its +# x16-sibling idles. +# +# WHY NOT FURTHER. One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or +# ~1.43 GiB (attention). The next step, 36:29, would cross the +# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB +# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the +# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap +# 0.58) is the farthest safe shift toward the healthy card at this pool. +# Do not move it without re-measuring. +TENSOR_SPLIT=35,30 NGL=999 # Auto-fit has no business resizing a context we set explicitly. diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index c4583c3..f68c895 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-medium llama.cpp Runbook (MTP drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj. +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj. tags: - frank-medium - dankserver @@ -145,7 +145,7 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 35:30 since 2026-09-17 (part 3): ROCm0 is CPU-root-port Gen4 x16, ROCm1 rides the chipset at Gen4 x1, and the drafter-bound card on the x1 path ran 98% against the x16 card's 40%. 36:29 fails both the 1 GiB floor (card 0 free 0.45) and the fit slack (gap 2.22) — this is the farthest safe shift | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | | `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section | | `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] | @@ -326,8 +326,10 @@ draft context is sized from the target, same law as the DFlash2 one) plus draft buffers the fit only estimated. The seam was recalibrated to the measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this -shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's -floor at this pool, so 34:31 is the only split that fits 2x196k. +shape — ROCm1's 1.08 GiB of headroom is real. 35:30 failed card 0's floor +only at the OLD 1.5 GiB bar (it clears 1.0 with 1.27 GiB to spare); 36:29 +fails both gates. Under the recalibrated 1.0 GiB floor 35:30 became +reachable — and the split went there the same day (part 3). The rope caveat deepens: the top 131k of a conversation (196608 vs the trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions @@ -339,6 +341,34 @@ Verified after deploy: `adding speculative implementation 'draft-mtp'` with present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len ~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`. +## 2026-09-17 (part 3): the PCIe topology, and the split leans to 35:30 + +`lspci -vv` on dankserver answers why the split now leans toward ROCm0: +the two cards do not sit on equal roads. + +| Card | Path | Bottleneck link | +|---|---|---| +| ROCm0 `03:00.0` | CPU root port `00:01.1` (Gen4 x16) -> switch -> card | Gen4 **x16** (~31.5 GB/s) | +| ROCm1 `0b:00.0` | CPU root port `00:02.1` (Gen4 x4) -> chipset -> `06:09.0` -> `09:00.0` -> card | Gen4 **x1** (~1.97 GB/s) | + +With the MTP drafter, the output layers and the heavier compute buffers +all pinned to the last-layers card, live `rocm-smi` under traffic read +**98% busy on the x1 card vs 40% on the x16 card**. The boundary crossing +itself is cheap (~5.2 MB of activation per 512-token ubatch rides the +wire — a few ms at 1.97 GB/s against a ~430 ms ubatch), but the +saturated card should not be the one behind the narrowest link. The +operator moved one layer to ROCm0: **34:31 -> 35:30**. The moved layer +is 34 — the 4k+2 position of block 8, a DeltaNet — which is what keeps +the shift cheap (~0.43 GiB). + +Fit at the new shape: card 0 30.59 GiB (free 1.27), card 1 30.01 GiB +(free 1.85), gap 0.58 — inside the 1.0 GiB floor and the 1.483 GiB +slack. The next step, 36:29, crosses the full-attention layer at 35 and +fails BOTH gates (card 0 free 0.45 GiB, gap 2.22 GiB), so the +layer-toward-ROCm0 lever is spent: further rebalancing means a ctx cut, +not a layer. The seam's balance test re-derives all of this from the env +file on every run. + ## Verified serving behaviour (2026-09-16) diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index c74a98f..727f3c2 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -158,7 +158,9 @@ print('draft-model wiring ok') # measured 6.44 / 6.90 GiB free. The fit reproduces card 1 to 0.00 GiB # and card 0 to 0.86 GiB at that point; it is a GATE, not a meter -- the # deployed point is re-measured on the host after every split/ctx/batch - # move (the current one is 34:31 / 393216 / MTP, 2026-09-17). + # move (the current one is 35:30 / 393216 / MTP, moved from 34:31 on + # 2026-09-17 to lean work off ROCm1's Gen4-x1 chipset path -- see the + # TENSOR_SPLIT header in frank-medium.env). # # card 0 = its share of the weights and of the KV (qwen35 spends KV on # its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim @@ -196,19 +198,20 @@ print('draft-model wiring ok') local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack local card_hi card_lo diff local mmproj=931145888 # mmproj-BF16.gguf, autoloaded onto the primary device - # 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216 - # / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the + # 1.5 GiB was the floor until the 2x196k shape: the 34:31 / 393216 / + # MTP point measured 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the # floor is the measured 1.08 GiB rounded down. The margin for one big - # allocation is thinner than it used to be; if a large allocation fails at - # load, the first move is a layer to ROCm0 (33:32), not a ctx cut. + # allocation is thinner than it used to be. The layer-toward-ROCm0 + # lever has since been spent (35:30, 2026-09-17): if a large allocation + # fails at load now, the next lever is a ctx cut, not another layer. floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card - # The deployed point (34:31 at ctx 393216 with MTP, the balance point at - # this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits - # above it at 1.483 GiB, which still rejects a one-layer move (worth - # ~0.43-1.43 GiB of gap depending on the boundary) while accepting the - # deployed split. (The 36:29 / 327680 / DFlash2 configuration's slack was - # 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the - # deployed shape.) + # The deployed point (35:30 at ctx 393216 with MTP) has a fit gap of + # 0.58 GiB -- closer to balanced than its 34:31 predecessor (1.06 GiB). + # The slack sits at 1.483 GiB: it accepts the deployed split and still + # rejects the next step toward ROCm0, 36:29, which fails BOTH gates + # (gap 2.22 GiB, card 0 free 0.45 GiB under the floor). (The 36:29 / + # 327680 / DFlash2 configuration's slack was 3225 MiB -- its 3.15 GiB + # gap was by construction, and is no longer the deployed shape.) slack=$(( 1483 * 1024 * 1024 )) card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \ + kv_pos * ctx * split_hi / (split_hi + split_lo) \