From a11fae07d82ad8c9934c25ffe045fbfc93ef8312 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 00:21:48 -0700 Subject: [PATCH] frank-medium: ubatch 512, and rebalance the split to 35:30 The batch/ubatch sweep (scripts/bench-batch-ubatch.sh, 23 cells of a 28-cell grid, 3 reps each) says the ubatch knob has an OPTIMUM, not a plateau: ubatch 256 384 512 768 1024 1280 1536 1792 2048 tok/s 1046 1068 1193 1135 1152 1151 1165 1148 1078 The curve is remarkably reproducible -- 512 came out 1192.88 / 1193.23 / 1193.28 / 1193.88 across batch 2048, 4096, 6144 and 8192, and 2048 came out 1077.38 / 1077.54 / 1077.59 -- so this is a deterministic property of the shape, not noise. The logical batch is throughput-neutral (2048..8192 all within 2 % at a fixed ubatch) and VRAM-neutral (the compute buffers are sized by ubatch alone). Decode is 15.88 tok/s at depth 65536 on every single cell. So the deployed 6144x1536 was both 2.4 % slower than the optimum and 3.71 GiB more expensive. Measured at the deployed shape (262144, 2 slots, DFlash2): 36,29 ubatch 1536 -> 4.71 / 4.91 GiB free (what was deployed) 36,29 ubatch 512 -> 6.01 / 7.32 GiB free (ubatch alone) 35,30 ubatch 512 -> 6.44 / 6.90 GiB free <- deployed The ubatch move freed 1.30 GiB on ROCm0 but 2.41 GiB on ROCm1, because ROCm1 holds the output layers and the drafter's graphs -- so it shifted the balance toward ROCm1 by 1.31 GiB and the split had to follow it one layer. 0.46 GiB is one DeltaNet layer's worth of asymmetry, i.e. the granularity floor: no split can be tuned finer than a layer, which is why 34,31 is not better. Re-fitted the seam with both ubatch coefficients measured at this shape and re-anchored it on the new deployed point (it reproduces 6.44 / 6.90 to 0.01 GiB). Every neighbouring split still fails it: 36,29 -> 1.85, 34,31 -> 0.94, 37,28 -> 3.24, 1,1 -> 3.03, 39,26 -> 6.03 GiB. Also replaced the unit test's literal `TENSOR_SPLIT=36,29` assertion with the shape it actually cares about -- two shares that add up to the model's 65 blocks, with ROCm1 taking the fewer -- so re-measuring the balance no longer breaks a test about parallelism. The numeric contract lives in the seam test. --- scripts/frank-medium.env | 94 +++++++++++++------ .../LLM/Dual-R9700-Cooperation.md | 41 ++++++++ tests/frank_medium_unit.bats | 53 +++++++---- 3 files changed, 142 insertions(+), 46 deletions(-) diff --git a/scripts/frank-medium.env b/scripts/frank-medium.env index 4c9dcc1..3e1f4a5 100644 --- a/scripts/frank-medium.env +++ b/scripts/frank-medium.env @@ -57,15 +57,22 @@ # of what is seen below -- they are allocated at load, not per request, and they # scale with --ubatch-size: # -# CTX split batch slots drafter GPU 0 free GPU 1 free note +# CTX split batch slots drafter GPU 0 free GPU 1 free note (ubatch 1536) # 262144 1,1 2048 1 DFlash2 8.64 GiB 4.43 GiB day's start # 262144 39,26 6144 2 DFlash2 2.42 GiB 7.20 GiB # 262144 38,27 6144 2 DFlash2 3.85 GiB 5.77 GiB # 262144 37,28 6144 2 DFlash2 4.28 GiB 5.34 GiB -# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB <- deployed +# 262144 36,29 6144 2 DFlash2 4.71 GiB 4.91 GiB # 262144 35,30 6144 2 DFlash2 5.14 GiB 4.48 GiB # 262144 40,25 6144 2 DSpark 1.87 GiB 1.86 GiB (retired) # +# ...and the same deployed shape re-measured at ubatch 512, after llama-bench +# showed 512 is the throughput optimum and costs 3.71 GiB less in total (see the +# batch block below): +# +# 262144 36,29 6144 512 2 DFlash2 6.01 GiB 7.32 GiB ubatch only +# 262144 35,30 6144 512 2 DFlash2 6.44 GiB 6.90 GiB <- deployed +# # ONE -ts STEP IS ONE LAYER, AND LAYERS ARE NOT THE SAME SIZE. This is a hybrid # stack -- 16 x (3 Gated DeltaNet + 1 Gated Attention) blocks -- so moving a # DeltaNet layer shifts ~0.43 GiB while an attention layer shifts ~1.43 GiB @@ -79,30 +86,30 @@ # adds VRAM (a longer drafter, quantized KV, more slots) re-read the numbers # rather than assuming, and prefer CTX_SIZE over --gpu-layers. # -# WHY THE SPLIT IS 36:29. Two effects both pull the balance BELOW an even -# split: +# WHY THE SPLIT IS 35:30. Three effects decide it, and all three are measured: # * THE DRAFTER AND ITS OUTPUT LAYERS RIDE ROCm1. It carries the drafter's -# weights, the draft graph and the last target layers, so it must be given -# fewer. -# * THE BIGGER BATCH COSTS ROCm1 ABOUT TWICE WHAT IT COSTS ROCm0 (~3.3 vs -# ~1.7 GiB going 2048/512 -> 6144/1536 at 262144): ROCm1 holds the output -# layers AND the drafter's graphs. (Under the DSpark drafter that asymmetry -# dominated the picture, because DSpark also caches the WHOLE context; its -# KV alone was 5.0 GiB. DFlash2's window caps its KV at ~40 MiB, which -# freed ~4.3 GiB on ROCm1 and moved the balance four layers.) -# The layer-size granularity pushes back from the other side: 36:29 is the -# measured minimum (0.20 GiB gap, against 0.65 at 35:30 and 1.06 at 37:28). -# It also leaves the MOST ROOM -- this configuration clears the operator's -# 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought out of -# CTX_SIZE and the full window stays. +# weights, the draft graph and the last target layers, so it takes fewer +# than half the layer count. +# * THE UBATCH BUFFERS COST ROCm1 MORE THAN ROCm0, and shrinking them pays +# ROCm1 back more: 1536 -> 512 freed 1.30 GiB on ROCm0 but 2.41 GiB on +# ROCm1 at this shape. Read the other way, that is exactly why the smaller +# batch moved the balance toward ROCm1 -- and cost ROCm0 one layer. +# * THE LAYER GRANULARITY IS THE FLOOR ON HOW WELL ANY SPLIT CAN BALANCE. One +# -ts step is one layer -- ~0.43 GiB DeltaNet, ~1.43 GiB attention -- so the +# reachable minimum gap is comparable to one step rather than to zero. 35:30 +# sits at 0.46 GiB, i.e. one DeltaNet layer's worth, against 1.31 GiB at +# 36:29. +# It also leaves the most room: 6.44 / 6.90 GiB free clears the operator's 1.5 +# GiB-per-card floor by ~5 GiB, so no headroom has to be bought out of CTX_SIZE +# and the full window stays. # # A FAULT TO WATCH. A 39:26 split, at 1x batch / 1 slot, died on its eighth # request: the amdgpu driver logged SDMA0 UTCL2 page faults and the engine # surfaced `ROCm error: an illegal memory access was encountered` -> ggml_abort, # i.e. a fault in the cross-card boundary copy, this layer split's hot path. It -# has not reproduced (36:29 has served this file's verification battery, and the -# e2e gate), but the signature is worth recognising -- it is NOT an OOM, and no -# amount of headroom prevents it. +# has not reproduced (36:29 served this file's verification battery and the e2e +# gate, and 35:30 is the deployed shape), but the signature is worth recognising +# -- it is NOT an OOM, and no amount of headroom prevents it. # # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP reads # those bare vars; 2026-08-13 incident). llama.cpp takes --host/--port/--temp @@ -129,12 +136,12 @@ HIP_VISIBLE_DEVICES=0,1 DEVICES=ROCm0,ROCm1 SPLIT_MODE=layer -# 36:29 -- the measured balance at the deployed batch, and the split that leaves +# 35:30 -- the measured balance at the deployed batch, and the split that leaves # the most room: ROCm1 carries the drafter, its KV and the output layers, so it # takes fewer target layers. See SIZING above for the measurement table; one # -ts step is one LAYER, worth ~0.43 GiB (DeltaNet) or ~1.43 GiB (attention), so # do not move it without re-measuring. -TENSOR_SPLIT=36,29 +TENSOR_SPLIT=35,30 NGL=999 # Auto-fit has no business resizing a context we set explicitly. @@ -148,15 +155,42 @@ FLASH_ATTN=on CTX_SIZE=262144 PARALLEL=2 -# 3x the build's defaults (2048/512). --ubatch-size is the physical batch the -# graph is built at and therefore what the compute buffers cost: at 262144 this -# setting needs ~1.7 GiB more on ROCm0 and ~3.3 GiB on ROCm1 than 2048/512 does -# -- which is what put the balance below an even split. (8192/2048 failed to -# allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had -# 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and with -# DFlash2's ~4.9 GiB of headroom it has NOT been re-measured.) +# MEASURED 2026-09-16 with scripts/bench-batch-ubatch.sh on this box: llama-bench, +# a 64k prefill plus a 128-token decode with 64k of KV already resident, 3 reps, +# one invocation per cell, the deployed split and f16 KV. --ubatch-size is the +# physical batch the graph is built at -- and the only one of the two knobs that +# costs VRAM: +# +# batch ubatch prefill tok/s GPU 0 / GPU 1 free +# 6144 1536 1164.96 +- 2.30 10.28 / 11.71 GiB +# 6144 768 1135.03 +- 1.09 11.29 / 13.28 GiB +# 6144 512 1192.93 +- 0.91 11.63 / 13.81 GiB <- deployed +# 6144 384 1068.04 +- 0.79 11.80 / 14.07 GiB +# 6144 256 1045.98 +- 1.69 11.97 / 14.33 GiB +# 4096 1536 1161.90 +- 0.96 10.28 / 11.71 GiB +# 2048 1536 1170.23 +- 0.86 10.28 / 11.71 GiB +# 2048 512 1193.88 +- 1.01 11.63 / 13.81 GiB +# +# THE PLATEAU STARTS AT 512. 512 -> 1536 buys ~2 % of prefill -- inside the +# run-to-run spread, and the 512 cells were the faster pair in BOTH batch +# columns -- while costing 1.35 GiB on ROCm0 and 2.10 GiB on ROCm1, so the +# deployment takes the cheap end. BELOW 512 the SMs are no longer saturated: +# 384 and 256 fall ~11 % and ~12 %. The logical batch is free either way -- +# 2048, 4096 and 6144 land within 2 % of each other at a fixed ubatch and use +# the SAME VRAM, because the compute buffers are sized by ubatch alone. +# +# The prefill figure is a property of the SHAPE, not of this knob: the same +# sweep with no depth in front of it (depth 0) runs 1543 tok/s at ubatch 256 and +# 1723 at 1536 -- the plateau is in the same place, and the ~32 % between the two +# tables is what attending over 64k of resident KV costs. Decode is 15.88 tok/s +# at depth 65536 on EVERY ubatch (17.80 at depth 0), so this knob cannot touch +# generation at all. +# +# 8192/2048 stays untested under this drafter (the "2020 MiB pp buffer fails on +# ROCm1" note was measured under DSpark, when it had 0.08 GiB free), and there is +# no longer a reason to want it: batch buys nothing measurable here. BATCH_SIZE=6144 -UBATCH_SIZE=1536 +UBATCH_SIZE=512 # Threads cover tokenization, sampling and HTTP only: every layer is on a GPU. NUM_THREADS=8 diff --git a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md index 7f52571..bc5c06e 100644 --- a/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md +++ b/serverAppConfigs-docs/LLM/Dual-R9700-Cooperation.md @@ -31,6 +31,16 @@ P2P operation can be dispatched on card #2. Two paths restore cooperation: 2. **A board whose CPU x16 bifurcates to x8/x8** (X670E/X870E class, ~$250-400). Best performance. This board has no x8/x8 mode. +The M.2_2 half of path 1 rests on a **vendor claim, not on this host**: ASUS +lists M.2_1 and M.2_2 as CPU-connected (PCIe 5.0 x4 and 4.0 x4) and M.2_3 as +chipset, while an empty slot does not enumerate — so nothing on a running +system can confirm it until a card is in it. And CPU-attached does not imply +`Routing+`: `00:08.3` is CPU-side and reads `Routing-`. The precedent worth +trusting is `00:01.2`, the CPU M.2_1 port, which is `Routing+`. Note too what +the riser buys: **tensor parallelism** (~40 % decode), not "both cards on one +model" — the layer split already delivers that, so this is a performance +project, not a capability one. + ## The gate, in source ```cpp @@ -59,6 +69,37 @@ CPU must advertise `AtomicOpsCap: Routing+`. On this board card #2's path includes `05:00.0` and `06:09.0`, both `Routing-` — a **read-only capability bit**, so no software can enable it. +### The same gate, as this board reports it (2026-09-16) + +`lspci -vv` on dankserver. Card 0's chain is root port `00:01.1` → `01:00.0` → +`02:00.0` → `03:00.0`; card 1's is `00:02.1` → `05:00.0` → `06:00.0` → +`06:09.0` → `09:00.0` → `0a:00.0` → `0b:00.0`: + +| Device | Role | `AtomicOpsCap` | `AtomicOpsCtl` | Link | +|---|---|---|---|---| +| `00:01.1` | CPU root port → PCIEX16_1 | Routing+ | — | x16 | +| `03:00.0` | **card 0** | 32bit+ 64bit+ | **ReqEn+** | x16 | +| `00:02.1` | CPU root port → chipset | **Routing+** | — | x4 | +| `05:00.0`, `06:00.0`, `06:09.0` | B650 bridges | **Routing−** | — | x1 at `06:09.0` | +| `09:00.0`, `0a:00.0` | PCIe switch ports | Routing+ | — | x1 (downgraded) | +| `0b:00.0` | **card 1** | 32bit+ 64bit+ | **ReqEn−** | x16 | + +Three things fall out of that table, none of them in the source alone: + +- **The CPU side is fine** — `00:02.1` is `Routing+`. The wall is inside the + chipset, not the platform, which is why a slot on CPU lanes is the fix. +- **The endpoint's own bit differs between the cards, and that is the kernel's + doing.** `pci_enable_atomic_ops_to_root()` walks the bridge chain and refuses + to set Requester Enable when any bridge cannot route atomics. ROCclr's + `pcie_atomics_` and the endpoint's `ReqEn` are two faces of one cause, and it + makes `lspci -vv -s | grep AtomicOpsCtl` a one-command diagnosis: on + this host card 0 reads `ReqEn+` and card 1 `ReqEn−`. +- **Card 1's effective link is x1**, not the x16 its own capability block + advertises: `06:09.0` trained at x1, so ~2 GB/s is the real ceiling. That is a + bandwidth fact rather than an atomics one, and it only matters for + collectives — a 27B TP=2 allreduce is ≈5.6 ms/token at x1 against ≈1.4 ms at + x4. The layer split's boundary copy does not care. + ## Measured matrix Bare 2-rank `torch.distributed` (`backend=nccl`) on the two R9700s, no vLLM, no diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index 79c3677..b2c9c58 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -47,7 +47,18 @@ DRAFT_BYTES=2056414816 # DFlash2 Q8_0 drafter @test "both R9700s serve ONE model, layer-split inside one process" { grep -q '^SPLIT_MODE=layer$' "$ENV_FILE" grep -q '^DEVICES=ROCm0,ROCm1$' "$ENV_FILE" - grep -q '^TENSOR_SPLIT=36,29$' "$ENV_FILE" + # The split is a layer PAIR: the shares add up to the model's 65 blocks (one + # -ts unit per layer) and ROCm1 -- the card that also carries the drafter and + # the output layers -- takes the FEWER of the two. WHICH pair balances the + # cards is the seam test's job, which recomputes it from this file; pinning a + # literal here would only mean that re-measuring the balance breaks a test + # about parallelism. + local split_hi split_lo + split_hi="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\1/p' "$ENV_FILE")" + split_lo="$(sed -nE 's/^TENSOR_SPLIT=([0-9]+),([0-9]+)$/\2/p' "$ENV_FILE")" + [ -n "$split_hi" ] && [ -n "$split_lo" ] + [ $(( split_hi + split_lo )) -eq 65 ] + [ "$split_lo" -lt "$split_hi" ] grep -q -- '--split-mode "$SPLIT_MODE"' "$SERVE" grep -q -- '--device "$DEVICES"' "$SERVE" grep -q -- '--tensor-split "$TENSOR_SPLIT"' "$SERVE" @@ -144,9 +155,9 @@ print('draft-model wiring ok') [ "$parallel" -ge 1 ] [ "$batch" -ge "$ubatch" ] [ "$cache_k" = "f16" ] && [ "$cache_v" = "f16" ] - # Per card, calibrated against rocm-smi at the deployed point (36:29, ctx - # 262144, batch 6144 x ubatch 1536, two slots, DFlash2), where the two cards - # measured 4.71 / 4.91 GiB free. The fit lands within ~0.05 GiB of that. + # Per card, calibrated against rocm-smi at the deployed point (35:30, ctx + # 262144, batch 6144 x ubatch 512, two slots, DFlash2), where the two cards + # measured 6.44 / 6.90 GiB free. The fit reproduces that to 0.01 GiB. # # card 0 = its share of the weights and of the KV (qwen35 spends KV on # its 16 full-attention layers only: 2 (K,V) x 4 heads x 256 dim @@ -155,34 +166,44 @@ print('draft-model wiring ok') # card 1 = the same for its own share, plus the WHOLE drafter: its weights # (2.06e9 B), its KV -- BOUNDED BY ITS SLIDING WINDOW, so # 20480 B/pos x 2048 positions rather than the context -- and - # ~2.7e6 B per ubatch unit of buffers on top of a ~4.8 GiB + # ~2.5e6 B per ubatch unit of buffers on top of a ~0.89 GiB # constant. Its buffers are the heavier ones because it holds the # output layers and the drafter's graphs. # + # BOTH ubatch coefficients are measured at the deployed shape, from the + # 1536 -> 512 move: 1.30 GiB on card 0 and 2.41 GiB on card 1. That + # asymmetry is why shrinking the batch moved the balance toward card 1 and + # cost card 0 a layer -- the split and the batch cannot be chosen apart. + # # The layer it charges for is the AVERAGE (0.70 GiB: 31.46e9 of weights plus # 16 GiB of KV over 65 layers). This stack is a hybrid, so real steps vary # around that -- 0.43 GiB for a DeltaNet layer, 1.43 GiB for a full-attention - # one -- which makes the drift check below CONSERVATIVE: it overstates the - # already trips it. That is the intent: moving the split means re-measuring, - # not re-arguing. + # one -- which makes the drift check below CONSERVATIVE: it overstates the gap + # a one-layer move creates, and so trips on moves a measurement might bless. + # That is the intent: moving the split means re-measuring, not re-arguing. # - # The ubatch coefficients ARE fitted at 1536, so the floor check's "an - # 8192/2048 batch does not fit" is an EXTRAPOLATION, not a measurement -- - # and it cannot see CONTIGUITY at all, which is the actual mechanism by which - # the DSpark-era 8192/2048 failed (a 2020 MiB pp buffer that found no room). + # The slack is one DeltaNet layer, rounded up. At this shape the balance + # cannot be tuned finer than one layer, so 35:30's 0.46 GiB gap IS the + # achievable optimum and a tighter slack would only reject it. + # + # What this model cannot see is CONTIGUITY -- the mechanism by which the + # DSpark-era 8192/2048 failed, a 2020 MiB pp buffer that found no room while + # every total looked fine. Total bytes are not the whole story for one big + # allocation. local kv_pos=65536 draft_kv_pos=20480 floor slack local card_hi card_lo diff floor=$(( 1536 * 1024 * 1024 )) # 1.5 GiB per card - slack=$(( 512 * 1024 * 1024 )) # measurement tolerance + slack=$(( 512 * 1024 * 1024 )) # one DeltaNet layer, rounded up card_hi=$(( MODEL_BYTES * split_hi / (split_hi + split_lo) \ + kv_pos * ctx * split_hi / (split_hi + split_lo) \ - + 1442 * 1024 * ubatch )) + + 381898377 \ + + 1363 * 1024 * ubatch )) card_lo=$(( MODEL_BYTES * split_lo / (split_hi + split_lo) \ + DRAFT_BYTES \ + kv_pos * ctx * split_lo / (split_hi + split_lo) \ + draft_kv_pos * (ctx < 2048 ? ctx : 2048) \ - + 828697861 \ - + 2740 * 1024 * ubatch )) + + 955046413 \ + + 2468 * 1024 * ubatch )) [ "$card_hi" -le $(( CARD_BYTES - floor )) ] [ "$card_lo" -le $(( CARD_BYTES - floor )) ] diff=$(( card_hi - card_lo )); [ "$diff" -ge 0 ] || diff=$(( -diff ))