dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: shift the layer split to 35:30, leaning work onto the x16 card

Browse Source 
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s).  With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card.  Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
This commit is contained in:
Frank
2026-09-17 18:58:06 -07:00
parent d6bc245934
commit 6569c63ec4
4 changed files with 83 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+10 -3
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 |
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
@@ -335,8 +335,15 @@ systemd unit `scripts/frank-medium.service`) serving
(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
`SPEC_TYPE` change brings the faster chain back), in **one process across both
R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point
at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved
to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric:
ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1
(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the
drafter, output layers and heavier compute buffers pinned to the
last-layers card, the bottlenecked card was running 98% while the x16 card
sat at 40%. 36:29 would cross a full-attention layer and fails both the
1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this
pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
via the BF16 mmproj. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
scripts/frank-medium.env
+24 -6
View File
@@ -141,12 +141,30 @@ HIP_VISIBLE_DEVICES=0,1
DEVICES=ROCm0,ROCm1
SPLIT_MODE=layer
# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
# to ROCm0 than the 36:29 of the DFlash2/327680 configuration). One -ts step
# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
# move it without re-measuring.
TENSOR_SPLIT=34,31
# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card. The
# two R9700s do NOT have symmetric PCIe paths. ROCm0 (03:00.0) hangs off
# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
# (0b:00.0) rides the B650 chipset through bridges that each negotiate
# Gen4 x1 (~1.97 GB/s). Live rocm-smi showed the x1 card at 98% busy
# against 40% on the x16 card: the MTP drafter, the output layers and the
# heavier compute buffers are structurally pinned to the card holding the
# LAST layers (the draft context shares the target's weights there), so
# the bottlenecked card was doing most of the work. Moving one layer to
# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
# card the bigger share. The layer-split boundary itself is not the cost
# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
# card behind the x1 link should not be the saturated one while its
# x16-sibling idles.
#
# WHY NOT FURTHER. One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
# ~1.43 GiB (attention). The next step, 36:29, would cross the
# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
# 1.483 GiB slack) -- both gates fail. 35:30 (fit: 30.59 / 30.01, gap
# 0.58) is the farthest safe shift toward the healthy card at this pool.
# Do not move it without re-measuring.
TENSOR_SPLIT=35,30
NGL=999
# Auto-fit has no business resizing a context we set explicitly.
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+34 -4
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (MTP drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
tags:
- frank-medium
- dankserver
@@ -145,7 +145,7 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 35:30 since 2026-09-17 (part 3): ROCm0 is CPU-root-port Gen4 x16, ROCm1 rides the chipset at Gen4 x1, and the drafter-bound card on the x1 path ran 98% against the x16 card's 40%. 36:29 fails both the 1 GiB floor (card 0 free 0.45) and the fit slack (gap 2.22) — this is the farthest safe shift |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
@@ -326,8 +326,10 @@ draft context is sized from the target, same law as the DFlash2 one) plus
draft buffers the fit only estimated. The seam was recalibrated to the
measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's
floor at this pool, so 34:31 is the only split that fits 2x196k.
shape — ROCm1's 1.08 GiB of headroom is real. 35:30 failed card 0's floor
only at the OLD 1.5 GiB bar (it clears 1.0 with 1.27 GiB to spare); 36:29
fails both gates. Under the recalibrated 1.0 GiB floor 35:30 became
reachable — and the split went there the same day (part 3).
The rope caveat deepens: the top 131k of a conversation (196608 vs the
trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
@@ -339,6 +341,34 @@ Verified after deploy: `adding speculative implementation 'draft-mtp'` with
present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
## 2026-09-17 (part 3): the PCIe topology, and the split leans to 35:30
`lspci -vv` on dankserver answers why the split now leans toward ROCm0:
the two cards do not sit on equal roads.
| Card | Path | Bottleneck link |
|---|---|---|
| ROCm0 `03:00.0` | CPU root port `00:01.1` (Gen4 x16) -> switch -> card | Gen4 **x16** (~31.5 GB/s) |
| ROCm1 `0b:00.0` | CPU root port `00:02.1` (Gen4 x4) -> chipset -> `06:09.0` -> `09:00.0` -> card | Gen4 **x1** (~1.97 GB/s) |
With the MTP drafter, the output layers and the heavier compute buffers
all pinned to the last-layers card, live `rocm-smi` under traffic read
**98% busy on the x1 card vs 40% on the x16 card**. The boundary crossing
itself is cheap (~5.2 MB of activation per 512-token ubatch rides the
wire — a few ms at 1.97 GB/s against a ~430 ms ubatch), but the
saturated card should not be the one behind the narrowest link. The
operator moved one layer to ROCm0: **34:31 -> 35:30**. The moved layer
is 34 — the 4k+2 position of block 8, a DeltaNet — which is what keeps
the shift cheap (~0.43 GiB).
Fit at the new shape: card 0 30.59 GiB (free 1.27), card 1 30.01 GiB
(free 1.85), gap 0.58 — inside the 1.0 GiB floor and the 1.483 GiB
slack. The next step, 36:29, crosses the full-attention layer at 35 and
fails BOTH gates (card 0 free 0.45 GiB, gap 2.22 GiB), so the
layer-toward-ROCm0 lever is spent: further rebalancing means a ctx cut,
not a layer. The seam's balance test re-derives all of this from the env
file on every run.
## Verified serving behaviour (2026-09-16)
tests/frank_medium_unit.bats
+15 -12
View File
@@ -158,7 +158,9 @@ print('draft-model wiring ok')
# measured 6.44 / 6.90 GiB free. The fit reproduces card 1 to 0.00 GiB
# and card 0 to 0.86 GiB at that point; it is a GATE, not a meter -- the
# deployed point is re-measured on the host after every split/ctx/batch
# move (the current one is 34:31 / 393216 / MTP, 2026-09-17).
# move (the current one is 35:30 / 393216 / MTP, moved from 34:31 on
# 2026-09-17 to lean work off ROCm1's Gen4-x1 chipset path -- see the
# TENSOR_SPLIT header in frank-medium.env).
#
# card 0 = its share of the weights and of the KV (qwen35 spends KV on
# its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
@@ -196,19 +198,20 @@ print('draft-model wiring ok')
local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack
local card_hi card_lo diff
local mmproj=931145888 # mmproj-BF16.gguf, autoloaded onto the primary device
# 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216
# / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
# 1.5 GiB was the floor until the 2x196k shape: the 34:31 / 393216 /
# MTP point measured 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
# floor is the measured 1.08 GiB rounded down. The margin for one big
# allocation is thinner than it used to be; if a large allocation fails at
# load, the first move is a layer to ROCm0 (33:32), not a ctx cut.
# allocation is thinner than it used to be. The layer-toward-ROCm0
# lever has since been spent (35:30, 2026-09-17): if a large allocation
# fails at load now, the next lever is a ctx cut, not another layer.
floor=$(( 1024 * 1024 * 1024 )) # 1.0 GiB per card
# The deployed point (34:31 at ctx 393216 with MTP, the balance point at
# this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits
# above it at 1.483 GiB, which still rejects a one-layer move (worth
# ~0.43-1.43 GiB of gap depending on the boundary) while accepting the
# deployed split. (The 36:29 / 327680 / DFlash2 configuration's slack was
# 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the
# deployed shape.)
# The deployed point (35:30 at ctx 393216 with MTP) has a fit gap of
# 0.58 GiB -- closer to balanced than its 34:31 predecessor (1.06 GiB).
# The slack sits at 1.483 GiB: it accepts the deployed split and still
# rejects the next step toward ROCm0, 36:29, which fails BOTH gates
# (gap 2.22 GiB, card 0 free 0.45 GiB under the floor). (The 36:29 /
# 327680 / DFlash2 configuration's slack was 3225 MiB -- its 3.15 GiB
# gap was by construction, and is no longer the deployed shape.)
slack=$(( 1483 * 1024 * 1024 ))
card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
+ kv_pos * ctx * split_hi / (split_hi + split_lo) \