frank-medium: shift the layer split to 35:30, leaning work onto the x16 card

The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s).  With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card.  Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.

AGENTS.md

@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 
 | Target | Service | Runtime | Model | Port |
 |--------|---------|---------|-------|------|
-| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 |
+|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
 | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
 | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
 
@@ -335,8 +335,15 @@ systemd unit `scripts/frank-medium.service`) serving
 (29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
 the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
 `SPEC_TYPE` change brings the faster chain back), in **one process across both
-R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point
+R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved
-at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
+to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric:
+ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1
+(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the
+drafter, output layers and heavier compute buffers pinned to the
+last-layers card, the bottlenecked card was running 98% while the x16 card
+sat at 40%.  36:29 would cross a full-attention layer and fails both the
+1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this
+pool.  393216 ctx over **2 slots** of 196608, batch 6144x512, vision
 via the BF16 mmproj.  DSpark was
 put up in the drafter's place the same day and reverted: it declares no sliding
 window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four

scripts/frank-medium.env

@@ -141,12 +141,30 @@ HIP_VISIBLE_DEVICES=0,1
 DEVICES=ROCm0,ROCm1
 SPLIT_MODE=layer
 
-# 34:31 -- the balance point at the 393216 pool (fit gap 1.44 GiB; MTP's head
+# 35:30 -- operator call 2026-09-17: lean the work onto the x16 card.  The
-# and its KV ride ROCm1 with the last layers, so the split sends MORE layers
+# two R9700s do NOT have symmetric PCIe paths.  ROCm0 (03:00.0) hangs off
-# to ROCm0 than the 36:29 of the DFlash2/327680 configuration).  One -ts step
+# CPU root port 00:01.1 at Gen4 x16 end to end (~31.5 GB/s); ROCm1
-# is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so do not
+# (0b:00.0) rides the B650 chipset through bridges that each negotiate
-# move it without re-measuring.
+# Gen4 x1 (~1.97 GB/s).  Live rocm-smi showed the x1 card at 98% busy
-TENSOR_SPLIT=34,31
+# against 40% on the x16 card: the MTP drafter, the output layers and the
+# heavier compute buffers are structurally pinned to the card holding the
+# LAST layers (the draft context shares the target's weights there), so
+# the bottlenecked card was doing most of the work.  Moving one layer to
+# ROCm0 shifts ~0.43 GiB of DeltaNet weights (layer 34 is a DeltaNet --
+# the 4k..4k+2 position of block 8) off the x1 card and gives the healthy
+# card the bigger share.  The layer-split boundary itself is not the cost
+# (~5.2 MB of activation per 512-token ubatch rides the wire), but the
+# card behind the x1 link should not be the saturated one while its
+# x16-sibling idles.
+#
+# WHY NOT FURTHER.  One -ts step is one LAYER: ~0.43 GiB (DeltaNet) or
+# ~1.43 GiB (attention).  The next step, 36:29, would cross the
+# full-attention layer at 35: the seam fit puts card 0 at 31.41 GiB
+# (free 0.45 GiB, under the 1.0 GiB floor) with a 2.22 GiB gap (over the
+# 1.483 GiB slack) -- both gates fail.  35:30 (fit: 30.59 / 30.01, gap
+# 0.58) is the farthest safe shift toward the healthy card at this pool.
+# Do not move it without re-measuring.
+TENSOR_SPLIT=35,30
 
 NGL=999
 # Auto-fit has no business resizing a context we set explicitly.

serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md

@@ -1,7 +1,7 @@
 ---
 type: configuration
 title: frank-medium llama.cpp Runbook (MTP drafter)
-description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 34:31 split, a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
+description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the model's own MTP head (the z-lab DFlash2 sidecar kept on disk, one env change away), as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split (skewed toward the x16-slot card; see the 2026-09-17 part 3 section), a 393216-position KV pool (196608 per conversation over two slots), vision via the BF16 mmproj.
 tags:
   - frank-medium
   - dankserver
@@ -145,7 +145,7 @@ Deployed flags:
 
 | Flag | Value | Why |
 |---|---|---|
-| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `34,31` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 34:31 is the balance point at the 393216 pool (gap 1.44 GiB in the fit); 35:30 and 36:29 fail the 1 GiB floor on card 0 at this pool |
+| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one. 35:30 since 2026-09-17 (part 3): ROCm0 is CPU-root-port Gen4 x16, ROCm1 rides the chipset at Gen4 x1, and the drafter-bound card on the x1 path ran 98% against the x16 card's 40%. 36:29 fails both the 1 GiB floor (card 0 free 0.45) and the fit slack (gap 2.22) — this is the farthest safe shift |
 | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
 | `--ctx-size` / `--parallel` | `393216` / `2` | the pool over two slots of 196608; `--parallel` **divides** the context. 1.5x the trained 262144 window, and the GGUF has no rope scaling — see the 2026-09-17 section |
 | `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
@@ -326,8 +326,10 @@ draft context is sized from the target, same law as the DFlash2 one) plus
 draft buffers the fit only estimated. The seam was recalibrated to the
 measured value: the MTP buffer constant is now 2,901,231,913 B (the 4.20 GiB
 total MTP charge on card 1), and the floor dropped 1.5 → **1.0 GiB** for this
-shape — ROCm1's 1.08 GiB of headroom is real, and 35:30/36:29 fail card 0's
+shape — ROCm1's 1.08 GiB of headroom is real. 35:30 failed card 0's floor
-floor at this pool, so 34:31 is the only split that fits 2x196k.
+only at the OLD 1.5 GiB bar (it clears 1.0 with 1.27 GiB to spare); 36:29
+fails both gates. Under the recalibrated 1.0 GiB floor 35:30 became
+reachable — and the split went there the same day (part 3).
 
 The rope caveat deepens: the top 131k of a conversation (196608 vs the
 trained 262144… i.e. 393216/2 = 196608 per slot, so the top 131,072 positions
@@ -339,6 +341,34 @@ Verified after deploy: `adding speculative implementation 'draft-mtp'` with
 present; 207 drafts / 108 accepted in a text run (52 % acceptance, mean len
 ~1.7); vision, tools and `reasoning_effort` all pass; e2e 32/32; `NRestarts=0`.
 
+## 2026-09-17 (part 3): the PCIe topology, and the split leans to 35:30
+
+`lspci -vv` on dankserver answers why the split now leans toward ROCm0:
+the two cards do not sit on equal roads.
+
+| Card | Path | Bottleneck link |
+|---|---|---|
+| ROCm0 `03:00.0` | CPU root port `00:01.1` (Gen4 x16) -> switch -> card | Gen4 **x16** (~31.5 GB/s) |
+| ROCm1 `0b:00.0` | CPU root port `00:02.1` (Gen4 x4) -> chipset -> `06:09.0` -> `09:00.0` -> card | Gen4 **x1** (~1.97 GB/s) |
+
+With the MTP drafter, the output layers and the heavier compute buffers
+all pinned to the last-layers card, live `rocm-smi` under traffic read
+**98% busy on the x1 card vs 40% on the x16 card**. The boundary crossing
+itself is cheap (~5.2 MB of activation per 512-token ubatch rides the
+wire — a few ms at 1.97 GB/s against a ~430 ms ubatch), but the
+saturated card should not be the one behind the narrowest link. The
+operator moved one layer to ROCm0: **34:31 -> 35:30**. The moved layer
+is 34 — the 4k+2 position of block 8, a DeltaNet — which is what keeps
+the shift cheap (~0.43 GiB).
+
+Fit at the new shape: card 0 30.59 GiB (free 1.27), card 1 30.01 GiB
+(free 1.85), gap 0.58 — inside the 1.0 GiB floor and the 1.483 GiB
+slack. The next step, 36:29, crosses the full-attention layer at 35 and
+fails BOTH gates (card 0 free 0.45 GiB, gap 2.22 GiB), so the
+layer-toward-ROCm0 lever is spent: further rebalancing means a ctx cut,
+not a layer. The seam's balance test re-derives all of this from the env
+file on every run.
+
 
 ## Verified serving behaviour (2026-09-16)
 

tests/frank_medium_unit.bats

@@ -158,7 +158,9 @@ print('draft-model wiring ok')
     # measured 6.44 / 6.90 GiB free.  The fit reproduces card 1 to 0.00 GiB
     # and card 0 to 0.86 GiB at that point; it is a GATE, not a meter -- the
     # deployed point is re-measured on the host after every split/ctx/batch
-    # move (the current one is 34:31 / 393216 / MTP, 2026-09-17).
+    # move (the current one is 35:30 / 393216 / MTP, moved from 34:31 on
+    # 2026-09-17 to lean work off ROCm1's Gen4-x1 chipset path -- see the
+    # TENSOR_SPLIT header in frank-medium.env).
     #
     #   card 0 = its share of the weights and of the KV (qwen35 spends KV on
     #            its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim
@@ -196,19 +198,20 @@ print('draft-model wiring ok')
     local kv_pos=65536 draft_kv_pos=20480 mtp_kv_pos=4096 floor slack
     local card_hi card_lo diff
     local mmproj=931145888   # mmproj-BF16.gguf, autoloaded onto the primary device
-    # 1.5 GiB was the floor until the 2x196k shape: the deployed 34:31 / 393216
+    # 1.5 GiB was the floor until the 2x196k shape: the 34:31 / 393216 /
-    # / MTP point measures 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
+    # MTP point measured 28.91 / 30.78 GiB, leaving 2.95 / 1.08 GiB, so the
     # floor is the measured 1.08 GiB rounded down.  The margin for one big
-    # allocation is thinner than it used to be; if a large allocation fails at
+    # allocation is thinner than it used to be.  The layer-toward-ROCm0
-    # load, the first move is a layer to ROCm0 (33:32), not a ctx cut.
+    # lever has since been spent (35:30, 2026-09-17): if a large allocation
+    # fails at load now, the next lever is a ctx cut, not another layer.
     floor=$(( 1024 * 1024 * 1024 ))     # 1.0 GiB per card
-    # The deployed point (34:31 at ctx 393216 with MTP, the balance point at
+    # The deployed point (35:30 at ctx 393216 with MTP) has a fit gap of
-    # this pool) has a post-recalibration fit gap of ~1.0 GiB; the slack sits
+    # 0.58 GiB -- closer to balanced than its 34:31 predecessor (1.06 GiB).
-    # above it at 1.483 GiB, which still rejects a one-layer move (worth
+    # The slack sits at 1.483 GiB: it accepts the deployed split and still
-    # ~0.43-1.43 GiB of gap depending on the boundary) while accepting the
+    # rejects the next step toward ROCm0, 36:29, which fails BOTH gates
-    # deployed split.  (The 36:29 / 327680 / DFlash2 configuration's slack was
+    # (gap 2.22 GiB, card 0 free 0.45 GiB under the floor).  (The 36:29 /
-    # 3225 MiB -- its 3.15 GiB gap was by construction, and is no longer the
+    # 327680 / DFlash2 configuration's slack was 3225 MiB -- its 3.15 GiB
-    # deployed shape.)
+    # gap was by construction, and is no longer the deployed shape.)
     slack=$(( 1483 * 1024 * 1024 ))
     card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \
               + kv_pos * ctx * split_hi / (split_hi + split_lo) \