dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the ubatch optimum, the rebalance, and the live atomics chain

Browse Source 
New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per
cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full
batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and
the limitations — including the one that cost me two runs, that amdgpu does not
return ~26 GiB at once, so a reload after a kill fails with "unable to allocate
ROCm0 buffer" and reads exactly like a configuration that does not fit.

AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the
10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead
of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at
the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the
frank-medium section now say 35:30 and batch 6144x512.

Runbook: deployed flags, the split history with the ubatch-512 table, the fault
note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent
requests).

Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1
ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser
premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not
imply Routing+ on this board (00:08.3 is CPU-side and Routing-).

log.md: the round trip recorded, newest first.
This commit is contained in:
Frank
2026-09-17 00:27:59 -07:00
parent a11fae07d8
commit 6472d706be
6 changed files with 275 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+21 -12
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 |
| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
@@ -321,9 +321,9 @@ systemd unit `scripts/frank-medium.service`) serving
`ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL**
(29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the
draftless **ngram-mod** matcher, in **one process across both R9700s**,
`--split-mode layer` at a **36:29 tensor split** (the balance point for the
`--split-mode layer` at a **35:30 tensor split** (the balance point for the
deployed batch: the drafter rides ROCm1, so it takes fewer target layers),
262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was
262144 ctx over **2 slots** of 131072, batch 6144x512, text only. DSpark was
put up in the drafter's place the same day and reverted: it declares no sliding
window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four
layers of the split.
@@ -551,15 +551,24 @@ linked runbooks.
trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
36:29), which is the practical size of the effect. See
[[frank-medium-llamacpp-dflash2]].
- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on
the card that holds the output layers and the drafter.** On frank-medium
(2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7,
which left the draft-carrying card at 0.08 GiB free under a split that was
balanced at the smaller batch. A split balanced at one batch is unbalanced at
another, so re-measure the balance after changing the batch. (8192/2048 failed
to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1
had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and
it has not been re-measured since.)
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
compute buffers land about TWICE as heavy on the card that holds the output
layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
shape (262144 over two slots, a 64k prefill measured at 64k depth,
`scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
`--batch-size` does not matter: 2048 through 8192 land within 2 % of each
other at a fixed ubatch and use identical VRAM. Decode never moves at all
(15.88 tok/s at depth 65536 on every cell). Because the buffers are
asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
Re-measure the balance after changing the batch. (8192/2048 failed to allocate
its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
model — the sweep's or the seam's — can see it.)
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md
+165
View File
@@ -0,0 +1,165 @@
---
type: reference
title: frank-medium — Batch and Ubatch Sweep
description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides.
tags:
- benchmark
- llamacpp
- ubatch
- frank-medium
- dankserver
- layer-split
---
# frank-medium — Batch and Ubatch Sweep
Run date: 2026-09-16. Harness: `scripts/bench-batch-ubatch.sh` (one
`llama-bench` invocation per cell, a VRAM sampler per cell, the service stopped
for the duration and restarted on exit). Tested at the deployed shape: 262144
ctx, two slots, `--split-mode layer`, f16 KV, flash attention on, DFlash2
loaded by the service but absent from the bench.
The question was how large the logical batch and the micro-batch may be, and
what each buys. The answer is that `--ubatch-size` has an **optimum** on this
stack, that the logical batch does not matter at all, and that the optimum is
much cheaper than what was deployed — which then moved the layer split.
## The shape of the measurement
A cell is a **64k prefill measured with 64k of KV already resident**, plus a
128-token decode at that same depth:
```
llama-bench -m <UD-Q8_K_XL> -p 65536 -n 128 -d 65536 -b <b> -ub <ub> \
-ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \
-fa on -ctk f16 -ctv f16 -r 3 --no-warmup
```
`-d 65536` is what puts depth in front of the measurement: a `tg` instance runs
with `n_prompt = 0`, and an unfilled `-d` would measure a cold cache. The depth
fill itself is excluded from the timing (and cached across reps), so the number
is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape
rather than a cold start.
Three `llama-bench` properties shaped the harness, all of them traps:
- **No `-c`**: the context is `n_prompt + n_gen + n_depth`, so a bench can never
decide whether a configuration's buffers fit beside the *deployed* KV.
- **A test that cannot allocate calls `exit(1)`**, taking the process with it —
hence one invocation per cell.
- **`/` separates devices within a configuration and `,` separates sweep
values**, so `-ts 36,29` would have benched each card on its own.
## Results — prefill, 64k depth (tok/s)
| `--batch-size` | ub 256 | ub 384 | **ub 512** | ub 768 | ub 1024 | ub 1280 | ub 1536 | ub 1792 | ub 2048 |
|---|---|---|---|---|---|---|---|---|---|
| 2048 | — | — | **1193.9** | 1147.9 | 1151.9 | 1141.7 | 1170.2 | 1128.8 | 1077.5 |
| 4096 | — | — | **1193.3** | 1129.6 | 1151.8 | 1143.7 | 1161.9 | 1151.9 | 1077.4 |
| 6144 | 1046.0 | 1068.0 | **1192.9** | 1135.0 | 1151.9 | 1151.4 | 1165.0 | 1147.7 | 1077.6 |
| 8192 | — | — | **1193.2** | — | — | — | — | — | — |
Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. **Decode is
15.87–15.89 tok/s on every single cell**, and the VRAM columns depend only on
the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space).
## Results — the same shape with no depth in front of it
`N_DEPTH=0`, same 64k prompt, two points for contrast:
| `--batch-size` | ubatch | prefill (tok/s) | decode (tok/s) |
|---|---|---|---|
| 6144 | 256 | 1543.5 ± 13.4 | 17.80 |
| 6144 | 1536 | 1722.7 ± 6.2 | 17.80 |
The **ubatch curve has the same shape at both depths** (below 512 is slower),
and the two tables differ by a uniform ~32 %: that is what attending over 64k of
resident KV costs. Depth is a tax on every configuration, not something the
batch knobs can trade against.
## What the data says
- **`--ubatch-size` peaks at 512 and loses on both sides.** Below it the SMs are
not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the
peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170.
- **The curve is deterministic, not noisy.** 512 came out 1192.88 / 1193.23 /
1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 /
1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three
independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap
between 512 and 1536 real rather than run-to-run drift.
- **The logical `--batch-size` is irrelevant here.** At a fixed ubatch, 2048,
4096, 6144 and 8192 land within 2 % of each other — and use *identical* VRAM,
because the compute buffers are sized by the ubatch alone.
- **Decode does not move at all.** 15.88 tok/s at 64k depth on every cell and
17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill.
## What it costs at the deployed shape
Prefill rates do not decide a deployment; VRAM does. The same shape
re-measured at 262144 over two slots (load, poll `/health`, read `rocm-smi`,
kill — no requests):
| Split | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---|---|---|---|
| 36,29 — deployed before | 1536 | 4.71 GiB | 4.91 GiB | 0.20 GiB |
| 36,29 | **512** | 6.01 GiB | 7.32 GiB | 1.31 GiB |
| **35,30 — deployed now** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46 GiB** |
Two things follow.
**The ubatch move is free money.** 1536 → 512 gives back 1.30 GiB on ROCm0 and
2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes *up*, not
down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0
does per unit of ubatch, because it holds the output layers and the drafter's
graphs.
**And it moves the balance, so the split had to follow.** Those uneven
give-backs left ROCm1 1.31 GiB lighter, and one layer is one `-ts` step: 36:29
became 35:30, which lands at 0.46 GiB — a *single DeltaNet layer's* worth of
asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB
per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old
configuration is not reachable at the new batch; 0.46 GiB is.
## The decision
```ini
BATCH_SIZE=6144 # throughput-neutral and VRAM-neutral; left at the old value
UBATCH_SIZE=512 # the measured optimum: fastest AND cheapest
TENSOR_SPLIT=35,30 # rebalanced for the new ubatch, measured not interpolated
```
Verified live after the deploy: `--tensor-split 35,30 --ubatch-size 512`, two
slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement to
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
## Limitations
- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
Its decode figure is the *drafterless* baseline (15.9 at 64k depth against the
service's ~44 with DFlash2 + ngram-mod), and it never exercises `--parallel` or
the 16 GiB KV of a real 262144 deployment. The throughput comparison between
batch values is sound; the absolute decode numbers are not the service's.
- **Five cells of the intended grid were not run** (batch 8192 above ubatch 512).
Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the
sweep was stopped deliberately once the curve was established.
- **A totals model cannot see contiguity.** The earlier DSpark-era failure at
8192/2048 was a 2020 MiB pp buffer that found no room while every total looked
fine. Nothing here rules that in or out; it only shows there is no reason to
want a batch that large.
- **A measurement like this needs a settle gate.** Re-loading the model
immediately after a kill fails with `unable to allocate ROCm0 buffer` — the
previous instance's ~26 GiB is not returned instantly — which reads exactly
like a configuration that does not fit. Wait for `rocm-smi` to show both cards
near-empty before the next load.
## See Also
- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split
history
- [[frank-small-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt
at depth 0 made prefill climb all the way to ubatch 2048. Depth is the
difference: with no KV to attend over, per-pass overhead dominates and a
bigger micro-batch wins; at 64k of resident KV it does not.
- [[Dual-R9700-Cooperation]] — why the cards are layer-split at all
- [[Dankserver System Specs]] — hardware inventory
serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md
+36 -20
View File
@@ -1,7 +1,7 @@
---
type: configuration
title: frank-medium llama.cpp Runbook (DFlash2 drafter)
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, 262144 ctx over two slots.
description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split, 262144 ctx over two slots.
tags:
- frank-medium
- dankserver
@@ -144,10 +144,10 @@ Deployed flags:
| Flag | Value | Why |
|---|---|---|
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one |
| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one |
| `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set |
| `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context |
| `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance |
| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] |
| `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB |
| `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision |
| `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer |
@@ -227,25 +227,38 @@ qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth
of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an
attention layer. Balance by measurement, never by interpolating.
36:29 is both the minimum and the point that leaves the most room: it clears the
operator's 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought
out of `CTX_SIZE` and the model's full 262144-token window stays.
**Then the batch moved, and the split followed it** (2026-09-16, the same day
— see [[frank-medium-batch-ubatch]]). 36:29 was the balance at ubatch **1536**;
the sweep showed 512 is both faster and 3.71 GiB cheaper, and — because the
compute buffers cost ROCm1 nearly twice what they cost ROCm0 — shrinking them
handed ROCm1 1.31 GiB more than it handed ROCm0:
| `-ts` | ubatch | GPU 0 free | GPU 1 free | gap |
|---|---:|---:|---:|---:|
| 36,29 | 1536 | 4.71 GiB | 4.91 GiB | 0.20 |
| 36,29 | 512 | 6.01 GiB | 7.32 GiB | 1.31 |
| **35,30 — deployed** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46** |
35:30's 0.46 GiB gap is one DeltaNet layer's worth, which is the floor here: no
split can be tuned finer than a layer, so the 0.20 GiB of the old configuration
is not reachable at this batch. It clears the operator's 1.5 GiB-per-card floor
by ~5 GiB, so no headroom is bought out of `CTX_SIZE` and the model's full
262144-token window stays.
A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not
be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure,
not a total-bytes one, and it has **not** been re-measured with DFlash2's ~4.9
GiB of headroom. (`tests/frank_medium_unit.bats` fits the deployed point and
predicts it still would not fit, but that is an extrapolation of a coefficient
fitted at ubatch 1536.)
not a total-bytes one, and the sweep has since removed the reason to care: with
the batch geometry measured, there is nothing to gain by making either knob
larger, so the question is closed rather than pending.
> **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its
> eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the
> engine surfaced `ROCm error: an illegal memory access was encountered` ->
> `ggml_abort` — a fault in the cross-card boundary copy, which is this layer
> split's hot path. It has not reproduced (36:29 has since served the whole
> verification battery and the e2e gate; 39:26 itself was only *loaded* at the
> deployed batch, not exercised), and it is **not** an OOM: no amount of
> headroom would prevent it.
> split's hot path. It has not reproduced (36:29 served the whole verification
> battery and the e2e gate, and 35:30 at ubatch 512 is the deployed shape now;
> 39:26 itself was only *loaded* at the deployed batch, not exercised), and it
> is **not** an OOM: no amount of headroom would prevent it.
## Verified serving behaviour (2026-09-16)
@@ -267,11 +280,11 @@ fitted at ubatch 1536.)
`draft_dflash` symbol with `block_size=7` and a Markov head, so read the
numbers, not the implementation name.
- Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch and split, `rocm-smi` reports **4.71 GiB free on GPU 0 and
4.91 GiB on GPU 1** of 31.86 GiB each — the same figures the balance search
predicted to 0.01 GiB. Under load (two concurrent requests, one per slot) they
sit at 4.43 / 4.61, a 0.18 GiB gap, and both are far above the operator's
1.5 GiB floor.
deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
measurement of the same shape to 0.01 GiB. Under load (two concurrent
requests, one per slot) they sit at **6.23 / 6.64**, and both are far above
the operator's 1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
@@ -444,7 +457,7 @@ produced wrong numbers on this fleet:
- **It separates devices differently from `llama-server`.** `/` splits devices
*within one configuration*, `,` splits *sweep values* — so this service's
`--tensor-split 36,29` is `-ts 36/29` to llama-bench, and the comma form
`--tensor-split 35,30` is `-ts 35/30` to llama-bench, and the comma form
silently benches each card on its own instead. Likewise `--device
ROCm0,ROCm1` → `-dev ROCm0/ROCm1`.
- **There is no `-c`.** The context is a consequence of the test shape:
@@ -518,3 +531,6 @@ gets the server defaults above; a client that sends its own wins. Text only.
- [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX
3090, MTP, the image path)
- [[Dankserver System Specs]] — hardware inventory
- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed
geometry and moved this split from 36:29 to 35:30: the ubatch optimum at 512,
why the logical batch is free, and the harness traps
serverAppConfigs-docs/LLM/index.md
+3
View File
@@ -113,6 +113,9 @@ frank-small, port 7771]
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
the split rebalance that followed it
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
on Dankclaw (retired)
serverAppConfigs-docs/index.md
+5
View File
@@ -191,6 +191,11 @@ The LLM inference fleet on the two hosts. Full index:
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-small-benchmark]] — the small-model benchmark suite
- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x
R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the
logical batch buys nothing measurable, decode never moves, and the ubatch's
uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved
the tensor split from 36:29 to 35:30
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
- [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes
serverAppConfigs-docs/log.md
+45
View File
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
# Bundle Change Log
## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30
- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`,
fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot
allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep
with it — a VRAM sampler per cell, and the service stopped for the run and
started again from a trap. It must run from a transient unit: the deploy's
`LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH
shell on these hosts only gets 8 MiB.
- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps):
prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated),
2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The
curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048
measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap
between 512 and the deployed 1536 real.
- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 %
of each other at a fixed ubatch and use identical VRAM, because the compute
buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at
64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform
~32 % of prefill.
- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster
and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch
buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB
for the 1024-unit move, measured at 262144 over two slots) and shrinking them
handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90
GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a
0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new
deployed point with both ubatch coefficients measured at this shape; every
neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 →
3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29`
assertion became a *shape* assertion — two shares summing to the model's 65
blocks with ROCm1 taking the smaller — so re-measuring the balance no longer
breaks a test about parallelism.
- **Measurement trap worth keeping**: re-loading immediately after a kill fails
with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at
once, and a 180 s settle is not enough; it reads exactly like a configuration
that does not fit. The measurement helper grew a wait-for-empty gate.
- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags
and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]]
gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0
against `ReqEn−` on card 1, because the kernel refuses the requester bit when
a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a
vendor claim rather than a fact.
## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so