docs: the ubatch optimum, the rebalance, and the live atomics chain

New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per
cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full
batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and
the limitations — including the one that cost me two runs, that amdgpu does not
return ~26 GiB at once, so a reload after a kill fails with "unable to allocate
ROCm0 buffer" and reads exactly like a configuration that does not fit.

AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the
10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead
of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at
the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the
frank-medium section now say 35:30 and batch 6144x512.

Runbook: deployed flags, the split history with the ubatch-512 table, the fault
note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent
requests).

Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1
ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser
premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not
imply Routing+ on this board (00:08.3 is CPU-side and Routing-).

log.md: the round trip recorded, newest first.
This commit is contained in:
Frank committed 2026-09-17 00:27:59 -07:00
1 parent a11fae07d8
commit 6472d706be
6 files changed
+275 -32

No files matched your search

+21 -12
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
@@ -321,9 +321,9 @@ systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
draftless **ngram-mod** matcher, in **one process across both R9700s**,
`--split-mode layer` at a **36:29 tensor split** (the balance point for the
`--split-mode layer` at a **35:30 tensor split** (the balance point for the
deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was
262144 ctx over **2 slots** of 131072, batch 6144x512, text only. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split.
@@ -551,15 +551,24 @@ linked runbooks.
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on
the card that holds the output layers and the drafter.** On frank-medium
(2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7,
which left the draft-carrying card at 0.08 GiB free under a split that was
balanced at the smaller batch. A split balanced at one batch is unbalanced at
another, so re-measure the balance after changing the batch. (8192/2048 failed
to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1
had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and
it has not been re-measured since.)
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
compute buffers land about TWICE as heavy on the card that holds the output
layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
shape (262144 over two slots, a 64k prefill measured at 64k depth,
`scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
`--batch-size` does not matter: 2048 through 8192 land within 2 % of each
other at a fixed ubatch and use identical VRAM. Decode never moves at all
(15.88 tok/s at depth 65536 on every cell). Because the buffers are
asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
Re-measure the balance after changing the batch. (8192/2048 failed to allocate
its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
model — the sweep's or the seam's — can see it.)
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one