dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-medium: shift the layer split to 35:30, leaning work onto the x16 card

Browse Source 
The two R9700s sit on unequal PCIe paths: ROCm0 (03:00.0) is CPU-root-port
Gen4 x16 (~31.5 GB/s); ROCm1 (0b:00.0) rides the B650 chipset through
bridges negotiating Gen4 x1 (~1.97 GB/s).  With the MTP drafter, output
layers and heavier compute buffers pinned to the last-layers card, live
rocm-smi read 98% busy on the x1 card vs 40% on the x16 card.  Moving one
DeltaNet layer (34) to ROCm0 shifts ~0.43 GiB off the bottlenecked card;
the seam fit lands 30.59/30.01 with gap 0.58 GiB, inside floor and slack.
36:29 fails both gates, so this lever is spent.
This commit is contained in:
Frank
2026-09-17 18:58:06 -07:00
parent d6bc245934
commit 6569c63ec4
4 changed files with 83 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+10 -3
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (34:31), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj) | 7770 |
|Dankserver|`frank-medium`|mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build)|ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL with the model's own MTP head (DFlash2 sidecar kept on disk, one env change away), 2x Radeon AI PRO R9700 in ONE layer-split process (35:30 — skewed toward the x16-slot card; the other sits behind a Gen4 x1 chipset path), 393216 ctx over 2 slots of 196608, batch 6144x512, vision (BF16 mmproj)|7770|
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, 1048576 pool / 4 slots of 262,144, q8_0 KV, the fleet's vision endpoint | 7771 |
@@ -335,8 +335,15 @@ systemd unit `scripts/frank-medium.service`) serving
(29.29 GiB) with the model's OWN **MTP head** (`blk.<n>.nextn.*` in the file;
the z-lab DFlash2 `Q8_0` sidecar stays on disk and in the fetch script — one
`SPEC_TYPE` change brings the faster chain back), in **one process across both
R9700s**, `--split-mode layer` at a **34:31 tensor split** (the balance point
at this pool), 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
R9700s**, `--split-mode layer` at a **35:30 tensor split** — one layer moved
to ROCm0 on 2026-09-17 because the cards' PCIe paths are not symmetric:
ROCm0 (03:00.0) is CPU-root-port Gen4 x16 (~31.5 GB/s) while ROCm1
(0b:00.0) rides the B650 chipset at Gen4 x1 (~1.97 GB/s), and with the
drafter, output layers and heavier compute buffers pinned to the
last-layers card, the bottlenecked card was running 98% while the x16 card
sat at 40%. 36:29 would cross a full-attention layer and fails both the
1 GiB floor and the fit slack, so 35:30 is the farthest safe shift at this
pool. 393216 ctx over **2 slots** of 196608, batch 6144x512, vision
via the BF16 mmproj. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four