docs: the ubatch optimum, the rebalance, and the live atomics chain
New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and the limitations — including the one that cost me two runs, that amdgpu does not return ~26 GiB at once, so a reload after a kill fails with "unable to allocate ROCm0 buffer" and reads exactly like a configuration that does not fit. AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the 10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the frank-medium section now say 35:30 and batch 6144x512. Runbook: deployed flags, the split history with the ubatch-512 table, the fault note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent requests). Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1 ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not imply Routing+ on this board (00:08.3 is CPU-side and Routing-). log.md: the round trip recorded, newest first.
This commit is contained in:
1 parent
a11fae07d8
commit
6472d706be
6 files changed
+275
-32
No files matched your search
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
|
||||
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 |
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
|
||||
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
|
||||
|
||||
@@ -321,9 +321,9 @@ systemd unit `scripts/frank-medium.service`) serving
|
||||
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
|
||||
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
|
||||
draftless **ngram-mod** matcher, in **one process across both R9700s**,
|
||||
`--split-mode layer` at a **36:29 tensor split** (the balance point for the
|
||||
`--split-mode layer` at a **35:30 tensor split** (the balance point for the
|
||||
deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
|
||||
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was
|
||||
262144 ctx over **2 slots** of 131072, batch 6144x512, text only. DSpark was
|
||||
put up in the drafter's place the same day and reverted: it declares no sliding
|
||||
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
|
||||
layers of the split.
|
||||
@@ -551,15 +551,24 @@ linked runbooks.
|
||||
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
|
||||
36:29), which is the practical size of the effect. See
|
||||
[[frank-medium-llamacpp-dflash2]].
|
||||
- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on
|
||||
the card that holds the output layers and the drafter.** On frank-medium
|
||||
(2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7,
|
||||
which left the draft-carrying card at 0.08 GiB free under a split that was
|
||||
balanced at the smaller batch. A split balanced at one batch is unbalanced at
|
||||
another, so re-measure the balance after changing the batch. (8192/2048 failed
|
||||
to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1
|
||||
had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and
|
||||
it has not been re-measured since.)
|
||||
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
|
||||
compute buffers land about TWICE as heavy on the card that holds the output
|
||||
layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
|
||||
shape (262144 over two slots, a 64k prefill measured at 64k depth,
|
||||
`scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
|
||||
at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
|
||||
1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
|
||||
SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
|
||||
ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
|
||||
`--batch-size` does not matter: 2048 through 8192 land within 2 % of each
|
||||
other at a fixed ubatch and use identical VRAM. Decode never moves at all
|
||||
(15.88 tok/s at depth 65536 on every cell). Because the buffers are
|
||||
asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
|
||||
1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
|
||||
Re-measure the balance after changing the batch. (8192/2048 failed to allocate
|
||||
its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
|
||||
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
|
||||
model — the sweep's or the seam's — can see it.)
|
||||
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
|
||||
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
|
||||
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
|
||||
|
||||
Reference in new issue
Block a user