docs(frank-small): record the batch doubling and scope the ubatch law to its backend

The AGENTS.md lesson "`--ubatch-size` has an OPTIMUM on this stack" was load-
bearing and over-general: it was measured on the 2x R9700 layer split, and the
same knob on the CUDA 3090 behaves the opposite way (curve climbs monotonically
to 2048). That difference is now the reason frank-small ships ubatch 2048, so
the lesson names the backend it was measured on and points at the counterpoint.

Adds the 2026-10-05 boot-pair A/B to the runbook (+4.6 % prefill at 8,928
tokens for 616 MiB, decode flat, the sub-ubatch prompt as the control), marks
the superseded 21,276/2,849 VRAM row, and logs the day. The old "ubatch 2048
failed the scratch boot" line stays visible as what it was: a predecessor-shape
observation, not a law.
This commit is contained in:
Frank
2026-10-05 15:53:06 -07:00
parent e5893dc30d
commit 67141c8c02
3 changed files with 107 additions and 7 deletions
@@ -68,12 +68,54 @@ were declined. Re-open only on evidence the Q6_K residual matters.
boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
contract*) and still prices every rollback shape.
## Compute-buffer geometry (2026-10-05)
`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
measured rather than assumed, because the predecessor shape recorded **"ubatch
2048 failed the scratch boot"** and this card has less free space than it did.
**That failure was a different shape, and it did not reproduce here.** The
service itself is the bench (one card holds one server, so no scratch port):
the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
2,849 MiB free — which makes the ubatch the only variable in the comparison.
|Probe|ubatch 1024|ubatch 2048|Δ|
|---|---|---|---|
|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
Three readings:
- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
than either micro-batch, so it cannot see the knob — and it did not move.
That is the control proving the 8.9k delta is geometry, not drift.
- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
prefill knob.
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
and the pool's 12,288-token (384 MiB) ladder in
[[frank-small-llama-mimo-q5km]] is the other way to buy room back.
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
ubatch verdict across backends.
The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
that is the pair the curve was measured at.
## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
|Check|Result|
|---|---|
|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung|
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|