diff --git a/AGENTS.md b/AGENTS.md index c446c46..9816b4a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-primary` | tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`, digest-pinned, plain `podman run`, root systemd unit) | `orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (~172.8 GiB, 131 shards), TP=2, expert offload automatic (fork plans it), prefetch OFF, **fp8 KV** (`--kv-cache-dtype fp8`), **no speculative decode**, MNBT 2048 (arena-free: the registry prices overhead flat up to its nbt 4096 anchor), MoE gather 32×64, **`--expert-precise-memory`** (fitted 5.25 GiB overhead, not the 7.60 GiB default), **`VLLM_CUDAGRAPH_MEM_BASE_MB=0` / `_PER_GRAPH_MB=12`** (the plan's CUDA-graph term priced at the measured capture — 0.31 GiB then a NEGATIVE −0.06 GiB delta, so the 256 MiB analytic base was fiction → graph term 0.07 GiB), **`GPU_MAX_HW_QUEUES=2`**, **mounted cache dir** at `/root/.cache` (the image ships no `/root/.cache` at all; unmounted, every boot re-pays `torch.compile took 181.20 s`), **3 slots × 200704 ctx** at KV_BYTES 3,166,500,000/rank (~400,700 KV tokens at the engine's 7,891.8 B/token here) — the **pool, not the slot count, caps resident conversations** (measured live: `GPU KV cache size: 401,408 tokens, Maximum concurrency for 200,704 tokens per request: 2.00×`; the 3.15e9 pool below it measured 398,658/1.99×), and shorter requests use the spare blocks. **`--max-num-seqs` must stay 3: SEQS=2 cannot boot this model** — torch.fx `ConstraintViolationError` on `query_start_loc`, measured at BOTH MNBT 1024 and 2048. **util stays 0.982 BY LAW**: at `le=1` the worker init refuses `budget == total` with `ValueError: Free memory on device cuda:N (31.29/31.86 GiB)` on both ranks before a single weight loads — measured crash loop 2026-10-03. Arena at shape G: **17.77 GiB / 7,762 slots measured**, KV 401,408 tokens / 2.00× concurrency, 0.89 GiB of real free VRAM per card held through a live 4,096-token generation (the ~3.5 GiB of driver-reported free VRAM at shape E was estimator slack — the registry's 9190 B/token KV reserve vs the real 7,892, and the 5.25 GiB overhead vs far less — NOT allocatable), and the client contract wired: `--reasoning-parser … | Dankclaw | `frank-large` | **ROCmFPX llama.cpp fork** (`kingjones30/ROCmFPX` @ `36e9acd`, gfx1151, built by `scripts/build-rocmfpx.sh`; bare `llama-server`, root systemd unit) | `kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF` **`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`** (23.15 GiB, ftype 114 ≈ 7.44 bpw: 353 F32 + 159 `Q8_0_ROCMFPX` + 339 `GGML_TYPE_Q6_0_ROCMFPX` tensors — only this fork can read them — with 64 protected output-head tensors), **MTP spec decode from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf` (n-max 4, pinned to ROCm0)**, **1 session × 262,144 (native window)** with **f16 KV**, the BF16 mmproj for vision, thinking on with the card's recipe (temp 1.0 / top_p 0.95 / top_k 20). Measured 2026-10-03: prose decode **27.6–28.1 t/s** at 4,813-token depth (draft acceptance 396/456 = 0.868, mean acceptance length ≈ 4.4), schema-echo 31.4 t/s at 100 %; `ngram-mod` chained onto MTP measured **−15 % on prose** and was rejected (the chain disables the hybrid's recurrent-state rollback). Cutover 2026-10-03 from the aligned Q8 file (17.0 t/s, acceptance 0.462 — rollback payload still on disk) | 7770 | -| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, 21,276 used / **2,849 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | +| Dankclaw | `frank-small` | llama.cpp master, CUDA sm_86 build (root systemd unit, relinked) | bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF **Q6_K** (9B dense qwen3_5 hybrid, 7.26 GiB GGUF, no MTP weights so no spec decode) + the repo's f16 mmproj for **vision**, on the RTX 3090, **2 × 200,704** (pool 401,408 — NOT the native window) with **fully unquantized KV (f16 K + f16 V)** under flash attention, **batch 4096 / ubatch 2048** (doubled 2026-10-05: measured +4.6 % prefill at 8,928 tokens for 616 MiB), 21,846 used / **2,279 free** of 24,576 MiB, 94.4 tok/s decode. Resized 2026-10-03 (3 × 180,224 + q8_0 V → 2 × 200,704 + f16 V → + Q6_K rung: the freed slots paid for exact values AND the higher quant; the rung costs a measured −10 % decode — weight-bandwidth-bound); rollback `MODEL_FILE=…/MiMo-Q5KM/…Q5_K_M.gguf` (buys the 10 % back) or `CTX_SIZE=540672 PARALLEL=3 CACHE_TYPE_V=q8_0` (Q5 only) | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -413,10 +413,18 @@ drives that term. **The shape (2026-10-03):** `--ctx-size 401408 --parallel 2` = **TWO conversations of 200,704 tokens each** (llama.cpp divides the pool: `n_ctx_slot = CTX_SIZE / PARALLEL`), `--flash-attn on` (the seam pins it), -**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, batch -2048/ubatch 1024 (ubatch 2048 failed the scratch boot). Measured VRAM **21,276 -used / 2,849 free** of 24,576 MiB — inside the 2–3 GiB floor Frank asked for -(non-KV 8,732 MiB; f16 K+V is 32 KiB/token across the 8 attention layers; +**f16 K + f16 V**, all layers on the GPU, `--mmproj` for vision, and since +2026-10-05 **batch 4096 / ubatch 2048** (doubled; the boot-pair A/B is in the +runbook). Measured trade at 8,928 prompt tokens: prefill 4,024 → **4,209 t/s, ++4.6 %**, decode flat (91.9–93.0 vs 93.8–94.3 — noise), and a 279-token prompt, +which sits below either ubatch, did not move at all. The old "ubatch 2048 +failed the scratch boot" note belongs to the Q5_K_M predecessor and does NOT +generalize: this shape boots 2048 clean. Measured VRAM **21,846 used / 2,279 +free** of 24,576 MiB against **21,276 / 2,849** for the 1024 arm, reproduced +byte-identically the same hour — the doubling costs **616 MiB** and lands at +the BOTTOM of the 2–3 GiB floor Frank asked for (`4096/1536` is the middle rung +if that margin is ever wanted; non-KV 9,302 MiB, up from 8,732 by exactly the +compute buffers; f16 K+V is 32 KiB/token across the 8 attention layers; quote `nvidia-smi`'s `free` field, never `total − used` — the driver holds ~450 MiB of reserve the subtraction hides). The 2026-10-02 3 × 180,224 + q8_0-V shape stays one env edit away (`CTX_SIZE=540672 PARALLEL=3 @@ -814,7 +822,8 @@ linked runbooks. trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> 36:29), which is the practical size of the effect. See [[frank-medium-llamacpp-dflash2]]. -- **`--ubatch-size` has an OPTIMUM on this stack, not a plateau — and its +- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau — + and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its compute buffers land about TWICE as heavy on the card that holds the output layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed shape (262144 over two slots, a 64k prefill measured at 64k depth, @@ -832,6 +841,19 @@ linked runbooks. its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and no totals model — the sweep's or the seam's — can see it.) +- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so + frank-small ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the + live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and + ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 → + 2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a + 279-token prompt, which is below either ubatch, came back identical — that is + the mechanical proof the knob was the only variable, since a prompt shorter + than the ubatch cannot see it. The logical `--batch-size` was again free. And + the predecessor's "ubatch 2048 failed the scratch boot" was a shape + observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench + it by restarting the SERVICE through a transient `FRANK_SMALL_ENV_FILE` + drop-in, never a scratch port — one card holds one server. See + [[frank-small-llama-mimo-q6k-2x200704]]. - **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md index 44904a8..7f19eca 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q6k-2x200704.md @@ -68,12 +68,54 @@ were declined. Re-open only on evidence the Q6_K residual matters. boot-pair table lives in [[frank-small-llama-mimo-q5km]] (*Shape and KV contract*) and still prices every rollback shape. +## Compute-buffer geometry (2026-10-05) + +`BATCH_SIZE=4096 UBATCH_SIZE=2048` — both doubled from 2048/1024, and the trade +measured rather than assumed, because the predecessor shape recorded **"ubatch +2048 failed the scratch boot"** and this card has less free space than it did. + +**That failure was a different shape, and it did not reproduce here.** The +service itself is the bench (one card holds one server, so no scratch port): +the old rung was re-booted through a transient `FRANK_SMALL_ENV_FILE` drop-in +and reproduced the 2026-10-03 boot **byte-identically** — 21,276 MiB used / +2,849 MiB free — which makes the ubatch the only variable in the comparison. + +|Probe|ubatch 1024|ubatch 2048|Δ| +|---|---|---|---| +|Prefill, 8,928 tokens|3,972 / 4,077 t/s|**4,207 / 4,212 t/s**|**+4.6 %**| +|Decode, 128 tokens at that depth|91.87 / 93.01 t/s|93.81 / 94.27 t/s|noise| +|Prefill, 279 tokens|2,675.7 t/s|2,658–2,811 t/s|noise| +|VRAM used / free|21,276 / 2,849 MiB|**21,846 / 2,279 MiB**|**−616 MiB**| + +Three readings: + +- **The gain only exists above the old ubatch.** A 279-token prompt is shorter + than either micro-batch, so it cannot see the knob — and it did not move. + That is the control proving the 8.9k delta is geometry, not drift. +- **Decode is flat**, as every prior sweep on this card found. Ubatch is a + prefill knob. +- **The price is 616 MiB out of a 2.8 GiB margin.** 2,279 MiB free is still + inside the 2–3 GiB band, at its bottom edge. `4096/1536` is the middle rung, + and the pool's 12,288-token (384 MiB) ladder in + [[frank-small-llama-mimo-q5km]] is the other way to buy room back. + +**Why 2048 is right HERE while 512 was right on the R9700 sibling: the curve is +backend-specific.** The 2× R9700 layer split peaks at ubatch 512 and is 10 % +off that peak by 2048 ([[frank-medium-batch-ubatch]]); on this CUDA card the +curve climbs monotonically to 2048 (710 → 1,188 t/s on the 35B MoE sweep, +1,944 → 2,423 t/s on the Ornith 8k sweep, both with decode flat). Never carry a +ubatch verdict across backends. + +The logical `--batch-size` is VRAM- and throughput-neutral — two sweeps and this +A/B agree — so 4096 costs nothing and buys nothing by itself; it is here because +that is the pair the curve was measured at. + ## Measured on this shape (2026-10-03, Q6_K, 2 × 200,704, f16 KV) |Check|Result| |---|---| |Boot to healthy|model + projector resident in ~2.3 s (load log `0.02.27`), `n_slots = 2`, `n_ctx_slot = 200704`, `loaded multimodal model`, health 200; deploy e2e gate 33/33| -|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung| +|VRAM|**21,276 MiB used / 2,849 MiB free** of 24,576 (`nvidia-smi` `free`, steady state) — inside the 2–3 GiB floor with exact V AND the higher rung. Superseded 2026-10-05 by the batch doubling: **21,846 / 2,279** — see *Compute-buffer geometry*| |Decode, temp 0, no thinking|**94.44 tok/s** on the identical 506-token probe that measured 104.84 on Q5_K_M — **−10 %**. Decode on this card is weight-bandwidth-bound, and Q6_K streams ~10 % more bytes per token; the quant rung buys quality with decode rate, measurably| |Prefill|2,229.9 t/s (same shallow probe; rates are depth-dependent, not comparable across probes)| |Vision|512×512 solid-red PNG → **`Red`** (277 prompt tokens — the image really reached the tower). See the probe-size trap in *Client contract*| diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 47ca0f3..4e29e45 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -1,5 +1,41 @@ # Bundle Change Log +## 2026-10-05 +- **frank-small: compute-buffer pair doubled to 4096/2048, and the prefill win + measured rather than assumed.** Frank asked for the batch and ubatch doubled. + The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048 + failed the scratch boot"**, and this shape carries 616 MiB less free space + than that predecessor did. + - The A/B used the service itself as the bench (one card, one server — no + scratch port): the old rung was re-booted through a transient + `FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03 + boot **byte-identically** (21,276 used / 2,849 free), which is what makes + ubatch the only variable. Repo never saw a fake revert commit; scaffolding + staged to `~/.AGENTTRASH/20261005-frank-small-ubatch-ab/`. + - **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %, + for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 — + noise). The control that proves it is geometry and not drift: a 279-token + prompt sits below *both* micro-batches, so it cannot see the knob, and it + did not move (2,676 vs 2,658–2,811). + - **The predecessor's boot failure did not reproduce** — that was a shape + observation, not a law; this shape boots 2048 clean. And the fleet lesson + "**`--ubatch-size` has an optimum**" is now scoped to what it actually is: + true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the + CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected + — the curve is backend-specific and must never be carried across backends. + - The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE` + and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test + asserts the argv follows the shipped env plus the `ubatch <= batch` + invariant. Proven live both ways — retuning the env to 8192/4096 keeps it + green, inverting to 2048/4096 turns it red. (The old assertions only ever + exercised the serve-script *defaults*, because the test env never carried + these two keys.) + - `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom + edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs + flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a + second burst and ~90 s). Docs: + [[frank-small-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*). + ## 2026-10-03 - **frank-primary: shape F then shape G — MNBT 2048, the CUDA-graph plan term pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0