docs(frank-small): record the batch doubling and scope the ubatch law to its backend
The AGENTS.md lesson "`--ubatch-size` has an OPTIMUM on this stack" was load- bearing and over-general: it was measured on the 2x R9700 layer split, and the same knob on the CUDA 3090 behaves the opposite way (curve climbs monotonically to 2048). That difference is now the reason frank-small ships ubatch 2048, so the lesson names the backend it was measured on and points at the counterpoint. Adds the 2026-10-05 boot-pair A/B to the runbook (+4.6 % prefill at 8,928 tokens for 616 MiB, decode flat, the sub-ubatch prompt as the control), marks the superseded 21,276/2,849 VRAM row, and logs the day. The old "ubatch 2048 failed the scratch boot" line stays visible as what it was: a predecessor-shape observation, not a law.
This commit is contained in:
@@ -68,12 +68,54 @@ were declined. Re-open only on evidence the Q6_K residual matters.
|
||||
boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
|
||||
contract*) and still prices every rollback shape.
|
||||
|
||||
## Compute-buffer geometry (2026-10-05)
|
||||
|
||||
`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
|
||||
measured rather than assumed, because the predecessor shape recorded **"ubatch
|
||||
2048 failed the scratch boot"** and this card has less free space than it did.
|
||||
|
||||
**That failure was a different shape, and it did not reproduce here.** The
|
||||
service itself is the bench (one card holds one server, so no scratch port):
|
||||
the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
|
||||
and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
|
||||
2,849 MiB free — which makes the ubatch the only variable in the comparison.
|
||||
|
||||
|Probe|ubatch 1024|ubatch 2048|Δ|
|
||||
|---|---|---|---|
|
||||
|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
|
||||
|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
|
||||
|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
|
||||
|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
|
||||
|
||||
Three readings:
|
||||
|
||||
- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
|
||||
than either micro-batch, so it cannot see the knob — and it did not move.
|
||||
That is the control proving the 8.9k delta is geometry, not drift.
|
||||
- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
|
||||
prefill knob.
|
||||
- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
|
||||
inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
|
||||
and the pool's 12,288-token (384 MiB) ladder in
|
||||
[[frank-small-llama-mimo-q5km]] is the other way to buy room back.
|
||||
|
||||
**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
|
||||
backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
|
||||
off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
|
||||
curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
|
||||
1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
|
||||
ubatch verdict across backends.
|
||||
|
||||
The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
|
||||
A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
|
||||
that is the pair the curve was measured at.
|
||||
|
||||
## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)
|
||||
|
||||
|Check|Result|
|
||||
|---|---|
|
||||
|Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
|
||||
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung|
|
||||
|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
|
||||
|Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
|
||||
|Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
|
||||
|Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
|
||||
|
||||
@@ -1,5 +1,41 @@
|
||||
# Bundle Change Log
|
||||
|
||||
## 2026-10-05
|
||||
- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||||
measured rather than assumed.** Frank asked for the batch and ubatch doubled.
|
||||
The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
|
||||
failed the scratch boot"**, and this shape carries 616 MiB less free space
|
||||
than that predecessor did.
|
||||
- The A/B used the service itself as the bench (one card, one server — no
|
||||
scratch port): the old rung was re-booted through a transient
|
||||
`FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
|
||||
boot **byte-identically** (21,276 used / 2,849 free), which is what makes
|
||||
ubatch the only variable. Repo never saw a fake revert commit; scaffolding
|
||||
staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
|
||||
- **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
|
||||
for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
|
||||
noise). The control that proves it is geometry and not drift: a 279-token
|
||||
prompt sits below *both* micro-batches, so it cannot see the knob, and it
|
||||
did not move (2,676 vs 2,658–2,811).
|
||||
- **The predecessor's boot failure did not reproduce** — that was a shape
|
||||
observation, not a law; this shape boots 2048 clean. And the fleet lesson
|
||||
"**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
|
||||
true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
|
||||
CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
|
||||
— the curve is backend-specific and must never be carried across backends.
|
||||
- The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
|
||||
and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
|
||||
asserts the argv follows the shipped env plus the `ubatch <= batch`
|
||||
invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
|
||||
green, inverting to 2048/4096 turns it red. (The old assertions only ever
|
||||
exercised the serve-script *defaults*, because the test env never carried
|
||||
these two keys.)
|
||||
- `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
|
||||
edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
|
||||
flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
|
||||
second burst and ~90 s). Docs:
|
||||
[[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||||
|
||||
## 2026-10-03
|
||||
- **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
|
||||
pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
|
||||
|
||||
Reference in New Issue
Block a user