diff --git a/AGENTS.md b/AGENTS.md index 357c005..cb4c4dc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 262144 ctx over 2 slots, batch 6144x1536, text only | 7770 | +| Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (35:30), 262144 ctx over 2 slots, batch 6144x512, text only | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | | Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | @@ -321,9 +321,9 @@ systemd unit `scripts/frank-medium.service`) serving `ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` at **UD-Q8_K_XL** (29.29 GiB) with the **z-lab DFlash2** `Q8_0` drafter chained to the draftless **ngram-mod** matcher, in **one process across both R9700s**, -`--split-mode layer` at a **36:29 tensor split** (the balance point for the +`--split-mode layer` at a **35:30 tensor split** (the balance point for the deployed batch: the drafter rides ROCm1, so it takes fewer target layers), -262144 ctx over **2 slots** of 131072, batch 6144x1536, text only. DSpark was +262144 ctx over **2 slots** of 131072, batch 6144x512, text only. DSpark was put up in the drafter's place the same day and reverted: it declares no sliding window, so its draft KV spanned the whole context (5.0 GiB) and cost ROCm1 four layers of the split. @@ -551,15 +551,24 @@ linked runbooks. trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> 36:29), which is the practical size of the effect. See [[frank-medium-llamacpp-dflash2]]. -- **The compute buffers of a bigger `--ubatch-size` land about TWICE as heavy on - the card that holds the output layers and the drafter.** On frank-medium - (2026-09-16) ubatch 512 -> 1536 cost ROCm1 ~3.3 GiB against ROCm0's ~1.7, - which left the draft-carrying card at 0.08 GiB free under a split that was - balanced at the smaller batch. A split balanced at one batch is unbalanced at - another, so re-measure the balance after changing the batch. (8192/2048 failed - to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 - had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and - it has not been re-measured since.) +- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its + compute buffers land about TWICE as heavy on the card that holds the output + layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed + shape (262144 over two slots, a 64k prefill measured at 64k depth, + `scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068 + at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280, + 1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the + SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs + ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical + `--batch-size` does not matter: 2048 through 8192 land within 2 % of each + other at a fixed ubatch and use identical VRAM. Decode never moves at all + (15.88 tok/s at depth 65536 on every cell). Because the buffers are + asymmetric, a split balanced at one batch is unbalanced at another: 512 -> + 1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30. + Re-measure the balance after changing the batch. (8192/2048 failed to allocate + its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB + free. That was a CONTIGUITY failure, not a total-bytes one, and no totals + model — the sweep's or the seam's — can see it.) - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one diff --git a/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md new file mode 100644 index 0000000..9af9b33 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md @@ -0,0 +1,165 @@ +--- +type: reference +title: frank-medium — Batch and Ubatch Sweep +description: llama-bench sweep of --batch-size against --ubatch-size on frank-medium (2x Radeon AI PRO R9700, layer split) at the deployed context shape, and the rebalance it forced. Prefill peaks at ubatch 512 and falls on BOTH sides. +tags: + - benchmark + - llamacpp + - ubatch + - frank-medium + - dankserver + - layer-split +--- + +# frank-medium — Batch and Ubatch Sweep + +Run date: 2026-09-16. Harness: `scripts/bench-batch-ubatch.sh` (one +`llama-bench` invocation per cell, a VRAM sampler per cell, the service stopped +for the duration and restarted on exit). Tested at the deployed shape: 262144 +ctx, two slots, `--split-mode layer`, f16 KV, flash attention on, DFlash2 +loaded by the service but absent from the bench. + +The question was how large the logical batch and the micro-batch may be, and +what each buys. The answer is that `--ubatch-size` has an **optimum** on this +stack, that the logical batch does not matter at all, and that the optimum is +much cheaper than what was deployed — which then moved the layer split. + +## The shape of the measurement + +A cell is a **64k prefill measured with 64k of KV already resident**, plus a +128-token decode at that same depth: + +``` +llama-bench -m -p 65536 -n 128 -d 65536 -b -ub \ + -ngl 999 -dev ROCm0/ROCm1 -sm layer -ts 36/29 \ + -fa on -ctk f16 -ctv f16 -r 3 --no-warmup +``` + +`-d 65536` is what puts depth in front of the measurement: a `tg` instance runs +with `n_prompt = 0`, and an unfilled `-d` would measure a cold cache. The depth +fill itself is excluded from the timing (and cached across reps), so the number +is "prefill a 64k prompt into a 64k conversation", which is the agent-loop shape +rather than a cold start. + +Three `llama-bench` properties shaped the harness, all of them traps: + +- **No `-c`**: the context is `n_prompt + n_gen + n_depth`, so a bench can never + decide whether a configuration's buffers fit beside the *deployed* KV. +- **A test that cannot allocate calls `exit(1)`**, taking the process with it — + hence one invocation per cell. +- **`/` separates devices within a configuration and `,` separates sweep + values**, so `-ts 36,29` would have benched each card on its own. + +## Results — prefill, 64k depth (tok/s) + +| `--batch-size` | ub 256 | ub 384 | **ub 512** | ub 768 | ub 1024 | ub 1280 | ub 1536 | ub 1792 | ub 2048 | +|---|---|---|---|---|---|---|---|---|---| +| 2048 | — | — | **1193.9** | 1147.9 | 1151.9 | 1141.7 | 1170.2 | 1128.8 | 1077.5 | +| 4096 | — | — | **1193.3** | 1129.6 | 1151.8 | 1143.7 | 1161.9 | 1151.9 | 1077.4 | +| 6144 | 1046.0 | 1068.0 | **1192.9** | 1135.0 | 1151.9 | 1151.4 | 1165.0 | 1147.7 | 1077.6 | +| 8192 | — | — | **1193.2** | — | — | — | — | — | — | + +Per-cell standard deviations are 0.5–2.3 tok/s, i.e. under 0.2 %. **Decode is +15.87–15.89 tok/s on every single cell**, and the VRAM columns depend only on +the ubatch (2048 / 4096 / 6144 at a fixed ubatch report identical free space). + +## Results — the same shape with no depth in front of it + +`N_DEPTH=0`, same 64k prompt, two points for contrast: + +| `--batch-size` | ubatch | prefill (tok/s) | decode (tok/s) | +|---|---|---|---| +| 6144 | 256 | 1543.5 ± 13.4 | 17.80 | +| 6144 | 1536 | 1722.7 ± 6.2 | 17.80 | + +The **ubatch curve has the same shape at both depths** (below 512 is slower), +and the two tables differ by a uniform ~32 %: that is what attending over 64k of +resident KV costs. Depth is a tax on every configuration, not something the +batch knobs can trade against. + +## What the data says + +- **`--ubatch-size` peaks at 512 and loses on both sides.** Below it the SMs are + not saturated (256 → 1046, 384 → 1068); at the top end 2048 is 10 % off the + peak (1077). The middle is a shallow basin: 768–1792 sit at 1130–1170. +- **The curve is deterministic, not noisy.** 512 came out 1192.88 / 1193.23 / + 1193.28 / 1193.88 across four different batch values, 1024 came out 1151.77 / + 1151.91 / 1151.94, and 2048 came out 1077.38 / 1077.54 / 1077.59. Three + independent batch columns agreeing to 0.4 tok/s is what makes the 2.4 % gap + between 512 and 1536 real rather than run-to-run drift. +- **The logical `--batch-size` is irrelevant here.** At a fixed ubatch, 2048, + 4096, 6144 and 8192 land within 2 % of each other — and use *identical* VRAM, + because the compute buffers are sized by the ubatch alone. +- **Decode does not move at all.** 15.88 tok/s at 64k depth on every cell and + 17.80 at depth 0, to ±0.05. Whatever this knob does, it does it to prefill. + +## What it costs at the deployed shape + +Prefill rates do not decide a deployment; VRAM does. The same shape +re-measured at 262144 over two slots (load, poll `/health`, read `rocm-smi`, +kill — no requests): + +| Split | ubatch | GPU 0 free | GPU 1 free | gap | +|---|---|---|---|---| +| 36,29 — deployed before | 1536 | 4.71 GiB | 4.91 GiB | 0.20 GiB | +| 36,29 | **512** | 6.01 GiB | 7.32 GiB | 1.31 GiB | +| **35,30 — deployed now** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46 GiB** | + +Two things follow. + +**The ubatch move is free money.** 1536 → 512 gives back 1.30 GiB on ROCm0 and +2.41 GiB on ROCm1 — 3.71 GiB in total — and the mean throughput goes *up*, not +down. The asymmetry is the interesting part: ROCm1 pays nearly twice what ROCm0 +does per unit of ubatch, because it holds the output layers and the drafter's +graphs. + +**And it moves the balance, so the split had to follow.** Those uneven +give-backs left ROCm1 1.31 GiB lighter, and one layer is one `-ts` step: 36:29 +became 35:30, which lands at 0.46 GiB — a *single DeltaNet layer's* worth of +asymmetry, i.e. the floor on how well any split can balance this stack (0.43 GiB +per DeltaNet layer, 1.43 per attention layer). The 0.20 GiB gap of the old +configuration is not reachable at the new batch; 0.46 GiB is. + +## The decision + +```ini +BATCH_SIZE=6144 # throughput-neutral and VRAM-neutral; left at the old value +UBATCH_SIZE=512 # the measured optimum: fastest AND cheapest +TENSOR_SPLIT=35,30 # rebalanced for the new ubatch, measured not interpolated +``` + +Verified live after the deploy: `--tensor-split 35,30 --ubatch-size 512`, two +slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement to +0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on +both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors. + +## Limitations + +- **`llama-bench` cannot run the drafter, the slots or the deployed context.** + Its decode figure is the *drafterless* baseline (15.9 at 64k depth against the + service's ~44 with DFlash2 + ngram-mod), and it never exercises `--parallel` or + the 16 GiB KV of a real 262144 deployment. The throughput comparison between + batch values is sound; the absolute decode numbers are not the service's. +- **Five cells of the intended grid were not run** (batch 8192 above ubatch 512). + Batch 8192 at ubatch 512 matches the other batch columns to 0.4 tok/s, and the + sweep was stopped deliberately once the curve was established. +- **A totals model cannot see contiguity.** The earlier DSpark-era failure at + 8192/2048 was a 2020 MiB pp buffer that found no room while every total looked + fine. Nothing here rules that in or out; it only shows there is no reason to + want a batch that large. +- **A measurement like this needs a settle gate.** Re-loading the model + immediately after a kill fails with `unable to allocate ROCm0 buffer` — the + previous instance's ~26 GiB is not returned instantly — which reads exactly + like a configuration that does not fit. Wait for `rocm-smi` to show both cards + near-empty before the next load. + +## See Also + +- [[frank-medium-llamacpp-dflash2]] — the deployment this tuned, and its split + history +- [[frank-small-benchmark]] — the same sweep on a CUDA 3090, where a 16k prompt + at depth 0 made prefill climb all the way to ubatch 2048. Depth is the + difference: with no KV to attend over, per-pass overhead dominates and a + bigger micro-batch wins; at 64k of resident KV it does not. +- [[Dual-R9700-Cooperation]] — why the cards are layer-split at all +- [[Dankserver System Specs]] — hardware inventory diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index a27d28f..6953404 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -1,7 +1,7 @@ --- type: configuration title: frank-medium llama.cpp Runbook (DFlash2 drafter) -description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 36:29 split, 262144 ctx over two slots. +description: The ACTIVE frank-medium deployment since 2026-09-16 — mainline llama.cpp on Dankserver serving ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP at UD-Q8_K_XL with the z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, as ONE layer-split process across both Radeon AI PRO R9700 cards at a 35:30 split, 262144 ctx over two slots. tags: - frank-medium - dankserver @@ -144,10 +144,10 @@ Deployed flags: | Flag | Value | Why | |---|---|---| -| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `36,29` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one | +| `--split-mode` / `--device` / `--tensor-split` | `layer` / `ROCm0,ROCm1` / `35,30` | both cards, one model, no collective; the split is skewed because ROCm1 carries the drafter, and it is one LAYER of granularity — 0.43 GiB on a DeltaNet boundary, 1.43 GiB on an attention one | | `--gpu-layers` / `--fit` | `999` / `off` | everything on the cards; auto-fit must not resize a context we set | | `--ctx-size` / `--parallel` | `262144` / `2` | the model's whole window over two slots of 131072; `--parallel` **divides** the context | -| `--batch-size` / `--ubatch-size` | `6144` / `1536` | 3x the defaults; `--ubatch-size` is what the compute buffers cost, and it is what forced the rebalance | +| `--batch-size` / `--ubatch-size` | `6144` / `512` | measured: ubatch 512 is the prefill optimum *and* the cheapest point on the curve; the logical batch is throughput- and VRAM-neutral. See [[frank-medium-batch-ubatch]] | | `--ctx-checkpoints` / `--cache-ram` | `0` / `8192` | no per-slot KV snapshots (the build's default); the RAM-side prompt cache is capped at 8 GiB | | `--flash-attn` / cache types | `on` / `f16` K and V | an agentic checkpoint is the wrong place to spend KV precision | | `--spec-type` / `--spec-draft-n-max` | `draft-dflash,ngram-mod` / `7` | DFlash2 blocks are 8 tokens, so 7 drafts is the ceiling; ngram-mod rides the chain as a draftless second proposer | @@ -227,25 +227,38 @@ qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth of that table — 37,28 -> 38,27 moves a DeltaNet layer, 38,27 -> 39,26 an attention layer. Balance by measurement, never by interpolating. -36:29 is both the minimum and the point that leaves the most room: it clears the -operator's 1.5 GiB-per-card floor by 3.2 GiB, so no headroom has to be bought -out of `CTX_SIZE` and the model's full 262144-token window stays. +**Then the batch moved, and the split followed it** (2026-09-16, the same day +— see [[frank-medium-batch-ubatch]]). 36:29 was the balance at ubatch **1536**; +the sweep showed 512 is both faster and 3.71 GiB cheaper, and — because the +compute buffers cost ROCm1 nearly twice what they cost ROCm0 — shrinking them +handed ROCm1 1.31 GiB more than it handed ROCm0: + +| `-ts` | ubatch | GPU 0 free | GPU 1 free | gap | +|---|---:|---:|---:|---:| +| 36,29 | 1536 | 4.71 GiB | 4.91 GiB | 0.20 | +| 36,29 | 512 | 6.01 GiB | 7.32 GiB | 1.31 | +| **35,30 — deployed** | **512** | **6.44 GiB** | **6.90 GiB** | **0.46** | + +35:30's 0.46 GiB gap is one DeltaNet layer's worth, which is the floor here: no +split can be tuned finer than a layer, so the 0.20 GiB of the old configuration +is not reachable at this batch. It clears the operator's 1.5 GiB-per-card floor +by ~5 GiB, so no headroom is bought out of `CTX_SIZE` and the model's full +262144-token window stays. A batch of 8192/2048 failed here under DSpark — its 2020 MiB pp buffer could not be allocated on ROCm1, which had 0.08 GiB free. That was a *contiguity* failure, -not a total-bytes one, and it has **not** been re-measured with DFlash2's ~4.9 -GiB of headroom. (`tests/frank_medium_unit.bats` fits the deployed point and -predicts it still would not fit, but that is an extrapolation of a coefficient -fitted at ubatch 1536.) +not a total-bytes one, and the sweep has since removed the reason to care: with +the batch geometry measured, there is nothing to gain by making either knob +larger, so the question is closed rather than pending. > **A fault to watch.** A 39:26 split, at 1x batch and one slot, died on its > eighth request: the amdgpu driver logged SDMA0 UTCL2 page faults and the > engine surfaced `ROCm error: an illegal memory access was encountered` -> > `ggml_abort` — a fault in the cross-card boundary copy, which is this layer -> split's hot path. It has not reproduced (36:29 has since served the whole -> verification battery and the e2e gate; 39:26 itself was only *loaded* at the -> deployed batch, not exercised), and it is **not** an OOM: no amount of -> headroom would prevent it. +> split's hot path. It has not reproduced (36:29 served the whole verification +> battery and the e2e gate, and 35:30 at ubatch 512 is the deployed shape now; +> 39:26 itself was only *loaded* at the deployed batch, not exercised), and it +> is **not** an OOM: no amount of headroom would prevent it. ## Verified serving behaviour (2026-09-16) @@ -267,11 +280,11 @@ fitted at ubatch 1536.) `draft_dflash` symbol with `block_size=7` and a Markov head, so read the numbers, not the implementation name. - Both cards hold the model and they are **balanced**: at 262144 with the - deployed batch and split, `rocm-smi` reports **4.71 GiB free on GPU 0 and - 4.91 GiB on GPU 1** of 31.86 GiB each — the same figures the balance search - predicted to 0.01 GiB. Under load (two concurrent requests, one per slot) they - sit at 4.43 / 4.61, a 0.18 GiB gap, and both are far above the operator's - 1.5 GiB floor. + deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and + 6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy + measurement of the same shape to 0.01 GiB. Under load (two concurrent + requests, one per slot) they sit at **6.23 / 6.64**, and both are far above + the operator's 1.5 GiB floor. - Drafting is live on **both slots**, which is what a 2-slot deployment has to prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted / 70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and @@ -444,7 +457,7 @@ produced wrong numbers on this fleet: - **It separates devices differently from `llama-server`.** `/` splits devices *within one configuration*, `,` splits *sweep values* — so this service's - `--tensor-split 36,29` is `-ts 36/29` to llama-bench, and the comma form + `--tensor-split 35,30` is `-ts 35/30` to llama-bench, and the comma form silently benches each card on its own instead. Likewise `--device ROCm0,ROCm1` → `-dev ROCm0/ROCm1`. - **There is no `-c`.** The context is a consequence of the test shape: @@ -518,3 +531,6 @@ gets the server defaults above; a client that sends its own wins. Text only. - [[frank-small-qwen38-gsq-rco]] — the fleet's other llama.cpp endpoint (RTX 3090, MTP, the image path) - [[Dankserver System Specs]] — hardware inventory +- [[frank-medium-batch-ubatch]] — the batch/ubatch sweep that set the deployed + geometry and moved this split from 36:29 to 35:30: the ubatch optimum at 512, + why the logical batch is free, and the harness traps diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index b7dc24d..c343561 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -113,6 +113,9 @@ frank-small, port 7771] (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11) - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results +- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split + (2026-09-16): the ubatch optimum at 512, why the logical batch is free, and + the split rebalance that followed it - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 1755490..3396afa 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -191,6 +191,11 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-large-benchmark]] — backend/spec benchmark behind the old head - [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results - [[frank-small-benchmark]] — the small-model benchmark suite +- [[frank-medium-batch-ubatch]] — the 2026-09-16 batch/ubatch sweep on the 2x + R9700 layer split: prefill PEAKS at ubatch 512 and falls on both sides, the + logical batch buys nothing measurable, decode never moves, and the ubatch's + uneven VRAM give-back (1.30 GiB on card 0 against 2.41 GiB on card 1) moved + the tensor split from 36:29 to 35:30 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes - [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU - [[qwen3.8-strix-optimizations]] — Strix Halo tuning notes diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 83fa4ac..b2a5bfb 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,51 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30 +- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`, + fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot + allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep + with it — a VRAM sampler per cell, and the service stopped for the run and + started again from a trap. It must run from a transient unit: the deploy's + `LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH + shell on these hosts only gets 8 MiB. +- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps): + prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated), + 2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The + curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 / + 1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048 + measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap + between 512 and the deployed 1536 real. +- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 % + of each other at a fixed ubatch and use identical VRAM, because the compute + buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at + 64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform + ~32 % of prefill. +- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster + and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch + buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB + for the 1024-unit move, measured at 262144 over two slots) and shrinking them + handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90 + GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a + 0.46 GiB gap — one DeltaNet layer, which is the granularity floor here. +- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new + deployed point with both ubatch coefficients measured at this shape; every + neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 → + 3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29` + assertion became a *shape* assertion — two shares summing to the model's 65 + blocks with ROCm1 taking the smaller — so re-measuring the balance no longer + breaks a test about parallelism. +- **Measurement trap worth keeping**: re-loading immediately after a kill fails + with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at + once, and a 180 s settle is not enough; it reads exactly like a configuration + that does not fit. The measurement helper grew a wait-for-empty gate. +- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags + and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]] + gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0 + against `ReqEn−` on card 1, because the kernel refuses the requester bit when + a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a + vendor claim rather than a fact. + ## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired - The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so