docs(frank-small): record the batch doubling and scope the ubatch law to its backend

The AGENTS.md lesson "`--ubatch-size` has an OPTIMUM on this stack" was load- bearing and over-general: it was measured on the 2x R9700 layer split, and the same knob on the CUDA 3090 behaves the opposite way (curve climbs monotonically to 2048). That difference is now the reason frank-small ships ubatch 2048, so the lesson names the backend it was measured on and points at the counterpoint. Adds the 2026-10-05 boot-pair A/B to the runbook (+4.6 % prefill at 8,928 tokens for 616 MiB, decode flat, the sub-ubatch prompt as the control), marks the superseded 21,276/2,849 VRAM row, and logs the day. The old "ubatch 2048 failed the scratch boot" line stays visible as what it was: a predecessor-shape observation, not a law.
2026-10-05 15:53:06 -07:00
parent e5893dc30d
commit 67141c8c02
3 changed files with 107 additions and 7 deletions
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
 |--------|---------|---------|-------|------|
 | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser …
 | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 |
-| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 21,276 used / **2,849 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |
+| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 |

 Retired endpoints (`frank-npu`, and every prior runtime) are in
 the [runbook index](#runbook-index). The full swap history lives in
@@ -413,10 +413,18 @@ drives that term.
 **The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO
 conversations of 200,704 tokens each** (llama.cpp divides the pool:
 `n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it),
-**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch
-2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **21,276
-used / 2,849 free** of 24,576 MiB — inside the 2–3 GiB floor Frank asked for
-(non-KV 8,732 MiB; f16 K+V is 32 KiB/token across the 8 attention layers;
+**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since
+2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the
+runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s,
+4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt,
+which sits below either ubatch, did not move at all. The old "ubatch 2048
+failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT
+generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279
+free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced
+byte-identically the same hour — the doubling costs **616 MiB** and lands at
+the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung
+if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the
+compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers;
 quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds
 ~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 +
 q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3
@@ -814,7 +822,8 @@ linked runbooks.
  trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
  36:29), which is the practical size of the effect. See
  [[frank-medium-llamacpp-dflash2]].
- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its
+- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
+  and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
  compute buffers land about TWICE as heavy on the card that holds the output
  layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
  shape (262144 over two slots, a 64k prefill measured at 64k depth,
@@ -832,6 +841,19 @@ linked runbooks.
  its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
  free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
  model — the sweep's or the seam's — can see it.)
+- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
+  frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
+  live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
+  ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
+  2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
+  279-token prompt, which is below either ubatch, came back identical — that is
+  the mechanical proof the knob was the only variable, since a prompt shorter
+  than the ubatch cannot see it. The logical `--batch-size` was again free. And
+  the predecessor's "ubatch 2048 failed the scratch boot" was a shape
+  observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
+  it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE`
+  drop-in, never a scratch port — one card holds one server. See
+  [[frank-small-llama-mimo-q6k-2x200704]].
 - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
  size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
  worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
@@ -68,12 +68,54 @@ were declined. Re-open only on evidence the Q6_K residual matters.
 boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV
 contract*) and still prices every rollback shape.

+## Compute-buffer geometry (2026-10-05)
+
+`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade
+measured rather than assumed, because the predecessor shape recorded **"ubatch
+2048 failed the scratch boot"** and this card has less free space than it did.
+
+**That failure was a different shape, and it did not reproduce here.** The
+service itself is the bench (one card holds one server, so no scratch port):
+the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in
+and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used /
+2,849 MiB free — which makes the ubatch the only variable in the comparison.
+
+|Probe|ubatch 1024|ubatch 2048|Δ|
+|---|---|---|---|
+|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**|
+|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise|
+|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise|
+|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**|
+
+Three readings:
+
+- **The gain only exists above the old ubatch.** A 279-token prompt is shorter
+  than either micro-batch, so it cannot see the knob — and it did not move.
+  That is the control proving the 8.9k delta is geometry, not drift.
+- **Decode is flat**, as every prior sweep on this card found. Ubatch is a
+  prefill knob.
+- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still
+  inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung,
+  and the pool's 12,288-token (384 MiB) ladder in
+  [[frank-small-llama-mimo-q5km]] is the other way to buy room back.
+
+**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is
+backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 %
+off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the
+curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep,
+1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a
+ubatch verdict across backends.
+
+The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this
+A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because
+that is the pair the curve was measured at.
+
 ## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV)

 |Check|Result|
 |---|---|
 |Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33|
-|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung|
+|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*|
 |Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably|
 |Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)|
 |Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*|
@@ -1,5 +1,41 @@
 # Bundle Change Log

+## 2026-10-05
+- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win
+  measured rather than assumed.** Frank asked for the batch and ubatch doubled.
+  The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
+  failed the scratch boot"**, and this shape carries 616 MiB less free space
+  than that predecessor did.
+  - The A/B used the service itself as the bench (one card, one server — no
+    scratch port): the old rung was re-booted through a transient
+    `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
+    boot **byte-identically** (21,276 used / 2,849 free), which is what makes
+    ubatch the only variable. Repo never saw a fake revert commit; scaffolding
+    staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`.
+  - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
+    for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
+    noise). The control that proves it is geometry and not drift: a 279-token
+    prompt sits below *both* micro-batches, so it cannot see the knob, and it
+    did not move (2,676 vs 2,658–2,811).
+  - **The predecessor's boot failure did not reproduce** — that was a shape
+    observation, not a law; this shape boots 2048 clean. And the fleet lesson
+    "**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
+    true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
+    CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
+    — the curve is backend-specific and must never be carried across backends.
+  - The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
+    and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
+    asserts the argv follows the shipped env plus the `ubatch <= batch`
+    invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
+    green, inverting to 2048/4096 turns it red. (The old assertions only ever
+    exercised the serve-script *defaults*, because the test env never carried
+    these two keys.)
+  - `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
+    edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
+    flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
+    second burst and ~90 s). Docs:
+    [[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
+
 ## 2026-10-03
 - **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term
  pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0