docs: the ubatch optimum, the rebalance, and the live atomics chain
New page [[frank-medium-batch-ubatch]]: the sweep's method (one invocation per cell, a VRAM sampler per cell, 64k prefill measured AT 64k depth), the full batch x ubatch matrix, the depth-0 control, the deployed-shape VRAM table, and the limitations — including the one that cost me two runs, that amdgpu does not return ~26 GiB at once, so a reload after a kill fails with "unable to allocate ROCm0 buffer" and reads exactly like a configuration that does not fit. AGENTS.md: the batch lesson now carries the measured curve (the peak at 512, the 10 % loss at 2048, the irrelevant logical batch, the invariant decode) instead of the "~3.3 vs ~1.7 GiB" figure, which was measured at the bench's shape; at the deployed shape the asymmetry is 2.41 against 1.30. Fleet row and the frank-medium section now say 35:30 and batch 6144x512. Runbook: deployed flags, the split history with the ubatch-512 table, the fault note, and the verified VRAM (6.44 / 6.90 idle, 6.23 / 6.64 under two concurrent requests). Dual-R9700-Cooperation: the live capability table (card 0 ReqEn+ against card 1 ReqEn-, x1 effective link on card 1) and an honest label on the M.2_2 riser premise — it is ASUS's spec sheet, not a measurement, and CPU-attached does not imply Routing+ on this board (00:08.3 is CPU-side and Routing-). log.md: the round trip recorded, newest first.
This commit is contained in:
1 parent
a11fae07d8
commit
6472d706be
6 files changed
+275
-32
No files matched your search
@@ -11,6 +11,51 @@ timestamp: 2026-09-14
|
||||
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-09-16 (late) — frank-medium: ubatch's optimum is 512, and it moved the split to 35:30
|
||||
- **New harness, shipped before use** (`scripts/bench-batch-ubatch.sh`, `87845fc`,
|
||||
fixed in `77a07ff`): one `llama-bench` invocation per cell — a test that cannot
|
||||
allocate calls `exit(1)`, so one infeasible ubatch would take the whole sweep
|
||||
with it — a VRAM sampler per cell, and the service stopped for the run and
|
||||
started again from a trap. It must run from a transient unit: the deploy's
|
||||
`LimitMEMLOCK=infinity` is load-bearing for ROCm staging buffers and an SSH
|
||||
shell on these hosts only gets 8 MiB.
|
||||
- **The sweep** (23 of 28 cells, a 64k prefill measured *at* 64k depth, 3 reps):
|
||||
prefill **peaks at ubatch 512**. 256 and 384 lose ~11 % (SMs not saturated),
|
||||
2048 loses 10 %, and 768–1792 sit in a shallow basin 3–5 % below the peak. The
|
||||
curve is deterministic rather than noisy — 512 measured 1192.88 / 1193.23 /
|
||||
1193.28 / 1193.88 across batch 2048 / 4096 / 6144 / 8192, and ubatch 2048
|
||||
measured 1077.38 / 1077.54 / 1077.59 — which is what makes the 2.4 % gap
|
||||
between 512 and the deployed 1536 real.
|
||||
- **The logical `--batch-size` is irrelevant**: 2048 through 8192 land within 2 %
|
||||
of each other at a fixed ubatch and use identical VRAM, because the compute
|
||||
buffers are sized by ubatch alone. Decode never moves at all — 15.88 tok/s at
|
||||
64k depth on every cell, 17.80 at depth 0 — while depth itself costs a uniform
|
||||
~32 % of prefill.
|
||||
- **Deployed change** (`a11fae0`): `UBATCH_SIZE` 1536 → 512, which is both faster
|
||||
and 3.71 GiB cheaper, and `TENSOR_SPLIT` 36,29 → 35,30, because the ubatch
|
||||
buffers cost ROCm1 nearly twice what they cost ROCm0 (1.30 against 2.41 GiB
|
||||
for the 1024-unit move, measured at 262144 over two slots) and shrinking them
|
||||
handed ROCm1 1.31 GiB more than ROCm0. Live after the deploy: **6.44 / 6.90
|
||||
GiB free** (was 4.71 / 4.91), 6.23 / 6.64 under two concurrent requests, and a
|
||||
0.46 GiB gap — one DeltaNet layer, which is the granularity floor here.
|
||||
- The balance seam in `tests/frank_medium_unit.bats` was re-fitted at the new
|
||||
deployed point with both ubatch coefficients measured at this shape; every
|
||||
neighbouring split still fails it (36,29 → 1.85 GiB, 34,31 → 0.94, 37,28 →
|
||||
3.24, 1,1 → 3.03, 39,26 → 6.03). The unit test's literal `TENSOR_SPLIT=36,29`
|
||||
assertion became a *shape* assertion — two shares summing to the model's 65
|
||||
blocks with ROCm1 taking the smaller — so re-measuring the balance no longer
|
||||
breaks a test about parallelism.
|
||||
- **Measurement trap worth keeping**: re-loading immediately after a kill fails
|
||||
with `unable to allocate ROCm0 buffer`. amdgpu does not hand ~26 GiB back at
|
||||
once, and a 180 s settle is not enough; it reads exactly like a configuration
|
||||
that does not fit. The measurement helper grew a wait-for-empty gate.
|
||||
- Docs: [[frank-medium-batch-ubatch]] (new page), the runbook's deployed flags
|
||||
and split history, and AGENTS.md's batch lesson. [[Dual-R9700-Cooperation]]
|
||||
gained the live `AtomicOpsCap` / `AtomicOpsCtl` chain — `ReqEn+` on card 0
|
||||
against `ReqEn−` on card 1, because the kernel refuses the requester bit when
|
||||
a bridge cannot route atomics — and its M.2_2 riser premise is now labelled a
|
||||
vendor claim rather than a fact.
|
||||
|
||||
## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired
|
||||
- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48,
|
||||
gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so
|
||||
|
||||
Reference in new issue
Block a user