docs: the ubatch optimum, the rebalance, and the live atomics chain

New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per
cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full
batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and
the limitations — including the one that cost me two runs, that amdgpu does not
return ~26 GiB at once, so a reload after a kill fails with "unable to allocate
ROCm0 buffer" and reads exactly like a configuration that does not fit.

AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the
10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead
of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at
the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the
frank-medium section now say 35:30 and batch 6144x512.

Runbook: deployed flags, the split history with the ubatch-512 table, the fault
note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent
requests).

Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1
ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser
premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not
imply Routing+ on this board (00:08.3 is CPU-side and Routing-).

log.md: the round trip recorded, newest first.
This commit is contained in:
Frank committed 2026-09-17 00:27:59 -07:00
1 parent a11fae07d8
commit 6472d706be
6 files changed
+275 -32

No files matched your search

+45
View File
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
# Bundle Change Log
## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30
- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`,
fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot
allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep
with it — a VRAM sampler per cell, and the service stopped for the run and
started again from a trap. It must run from a transient unit: the deploy's
`LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH
shell on these hosts only gets 8 MiB.
- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps):
prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated),
2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The
curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 /
1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048
measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap
between 512 and the deployed 1536 real.
- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 %
of each other at a fixed ubatch and use identical VRAM, because the compute
buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at
64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform
~32 % of prefill.
- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster
and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch
buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB
for the 1024-unit move, measured at 262144 over two slots) and shrinking them
handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90
GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a
0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new
deployed point with both ubatch coefficients measured at this shape; every
neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 →
3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29`
assertion became a *shape* assertion — two shares summing to the model's 65
blocks with ROCm1 taking the smaller — so re-measuring the balance no longer
breaks a test about parallelism.
- **Measurement trap worth keeping**: re-loading immediately after a kill fails
with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at
once, and a 180 s settle is not enough; it reads exactly like a configuration
that does not fit. The measurement helper grew a wait-for-empty gate.
- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags
and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]]
gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0
against `ReqEn−` on card 1, because the kernel refuses the requester bit when
a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a
vendor claim rather than a fact.
## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so