From 17450dd172175e0bc8f4f5f2167d9bf90bbb576d Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 04:56:59 -0700 Subject: [PATCH] docs: the max-length sweep tabled; frank-small.env NGL/FLASH_ATTN restored - runbook: identity now 4x262144, audio verified, throughput table with the OOM and negative-result rows, deployed flags updated to reality. - AGENTS.md frank-small paragraph matches the deployed shape. - log.md: the sweep entry. - scripts/frank-small.env: my earlier line-range edit had clobbered NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The live service was unaffected because the script's defaults match the declared values exactly, but the file contract is restored. Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved. --- AGENTS.md | 24 +++--- scripts/frank-small.env | 5 ++ .../LLM/frank-small-gemma4-12b.md | 79 ++++++++++++++----- serverAppConfigs-docs/log.md | 28 +++++++ 4 files changed, 102 insertions(+), 34 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f94e93b..47f54c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -306,22 +306,20 @@ serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M. The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4 nextn layers) loaded via `--model-draft`; the draftless **ngram-mod** matcher chains behind it in the same `--spec-type` value. BF16 mmproj, -port 7771, q8_0 KV, `CTX_SIZE` unset so `--fit` sizes the pool, 1 slot, -greedy default. Deployed flags are in `scripts/frank-small.env`; the serve -script refuses to start if the weights, the projector, the sidecar or the -binary are missing. +port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4` +(four FULL 262,144-token slots — the measured card ceiling), batch +**8192/1024**, greedy default. Deployed flags are in +`scripts/frank-small.env`; the serve script refuses to start if the +weights, the projector, the sidecar or the binary are missing. Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a chat completion returns content plus `reasoning_content`; a tool call -returns `tool_calls`; the slot timing lines show `draft acceptance = -0.38–0.46, mean len ≈ 2.2` (MTP engaged); an image request is answered -from the projector. `--fit` took the FULL window — `n_slots = 1, -n_ctx_slot = 262144` — because Gemma4's sliding-window pattern keeps the -q8 KV pool cheap: 11,996 MiB of the card's 24,576 MiB in use. The -`Gemma4Assistant requires ctx_other to be set` error that prints at boot -is the fit's own probe (the log calls it normal); the sidecar then loads -against the real context. Gemma4's template accepts every -`reasoning_effort` level (all seven probed: 200). +returns `tool_calls`; an image request is answered from the projector; +an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at +max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft +acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why +ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll +100 is WORSE at long ctx) is tabled in the runbook. Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. Rollback: [[frank-small-qwen38-gsq-rco]]. diff --git a/scripts/frank-small.env b/scripts/frank-small.env index dd23504..42d8431 100644 --- a/scripts/frank-small.env +++ b/scripts/frank-small.env @@ -89,6 +89,11 @@ UBATCH_SIZE=1024 # -- this host's CPU is shared with halogen's engine, which is CPU-hungry. NUM_THREADS=8 THREAD_BATCH=8 + +# Every layer on the 3090, with flash attention (required for q8 KV). +NGL=999 +FLASH_ATTN=on + # Speculative decoding: the MTP sidecar (draft context) chained with the # draftless ngram-mod matcher. Draft depth and busy-wait level are made # explicit here (both were the implicit build defaults: 3 and 50). diff --git a/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md b/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md index 4958b68..1e7c0ff 100644 --- a/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md +++ b/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md @@ -30,8 +30,8 @@ timestamp: 2026-09-17 | model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M | | model dir | `/home/dank/models/Gemma4-12B-QAT/` | | port | 7771 (alias `frank-small`; `frank-large` keeps 7770) | -| window | `--fit on` → **262,144** ctx, 1 slot; q8_0 K and V | -| vision | fleet image path via BF16 mmproj (`input: [text, image]`) | +| window | **4 slots x 262,144** (`CTX_SIZE=1048576`, `PARALLEL=4`); q8_0 K and V | +| vision + audio | fleet image path via BF16 mmproj (`input: [text, image]`); `input_audio` transcribes verbatim (verified) | ## Payload (all sha256-verified against the Hub) @@ -54,25 +54,30 @@ timestamp: 2026-09-17 cosmetic only. - **mmproj**: clip, vision 224/16 patches, **also carries an audio encoder** (`has_audio_encoder=True`) — vision is the fleet contract, - audio is inert bonus. + and audio works: a JFK WAV sent as an `input_audio` content part was + transcribed verbatim (2026-09-17). - **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure assistant (no trunk). Its KV **shares** the target's last layers — the share lambda at `llama-model.cpp:2662` references `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the drafter therefore costs no second KV pool. -## KV arithmetic — why 262k fits +## KV arithmetic — why four full windows fit Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) = **16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the sliding layers cache only the window; the full-attention layers cache the -full context. The engine's own `--fit` measured the combination and took -the whole 262,144-token window: total footprint at idle is **11,996 MiB** -of the card's 24,576 MiB, weights included. +full context — the pattern that makes the window cheap (the naive +"all-global" worst case was 32 GiB at q8/262k, impossible). -This is the practical payoff of a sliding-window architecture: the naive -"all-global" worst case was 32 GiB at q8/262k — impossible; the real -pattern makes the full window the cheapest possible answer. +Measured on the 3090 (24,576 MiB): one full-context slot idles at +11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed. +A fifth slot projects past the card, so **4 x 262,144 is the ceiling**. +`CTX_SIZE` is set explicitly to `PARALLEL x 262144` rather than left for +`--fit`: fit maximises the TOTAL pool and would divide a smaller window +per slot at `parallel > 1`. The slot-grid seam in +`tests/frank_small_unit.bats` pins both the full-window division and the +budget inequality so an oversized bump fails pre-deploy, not in boot. ## Spec-decode chain @@ -91,16 +96,49 @@ $MODEL_DIR/mtp-gemma-4-12B-it.gguf`: much smaller model — the point of the swap was the VRAM and the window, not acceptance). +## Throughput tuning (measured 2026-09-17) + +Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn), +prefill ~254k tokens / decode at ~253k context: + +| batch (b/ub) | draft n-max / poll | pp t/s | tg@250k t/s | idle MiB | +|---|---|---|---|---| +| 2048/512 (old) | 3 / 50 | 912–924 | 90*–204 | 22,346 | +| **8192/1024 (deployed)** | **3 / 50** | **962–965** | **226–302** | **23,300** | +| 8192/1024 | 4 / 100 | 961 | 230–257 | 23,300 | +| 4096/1024 | 4 / 100 | 959 | 205–277 | 23,300 | +| 8192/2048 | — | — | — | OOM | + +- **ubatch 1024 is the winner**: +5% max-length prefill, and decode at + 250k clearly above the 512 baseline. +- **ubatch 2048 cannot run**: the MTP draft context needs ~3.3 GiB of pp + compute buffer at that ubatch against ~2.2 GiB headroom — it dies in + `common_speculative_init_result` after the main model loads. +- **Deeper draft and busy-wait HURT long-ctx decode**: draft n-max 4 with + poll 100 verified a longer round per token, and at 250k the extra + full-attention re-read outweighs the accepted-token gain (230–257 vs + 257–302). Keep 3 / 50. +- **All ubatch-1024 variants idle identically** (23,300 MiB) — compute + buffers are reserved at max graph size regardless of the logical batch, + so 8192 costs nothing over 4096. +- **Concurrency caveat**: a long prefill monopolises the GPU — a decode + running beside a 254k-token prefill drops from ~240 to ~100–135 t/s. + Four slots share one engine with no per-slot QoS; that is inherent, not + a config fault. +- **No CUDA graphs in this build** — `--help` shows zero graph flags; the + decode-launch-overhead lever does not exist here. + ## Deployed flags Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`. Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`, `MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, `DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, -`SPEC_TYPE=draft-mtp,ngram-mod`, `CTX_SIZE` **unset** (→ `--fit on`), -`PARALLEL=1`, `NGL=999`, `FLASH_ATTN=on`, `--jinja`, `--metrics`, -greedy `--temp 0.0` server default. The serve script refuses to start on -any missing file. +`SPEC_TYPE=draft-mtp,ngram-mod`, `SPEC_DRAFT_N_MAX=3`, `POLL_LEVEL=50`, +`CTX_SIZE=1048576` with `PARALLEL=4` (four full 262,144-token slots), +`BATCH_SIZE=8192`, `UBATCH_SIZE=1024`, `NGL=999`, `FLASH_ATTN=on`, +`--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve +script refuses to start on any missing file. ## Client contract @@ -112,17 +150,16 @@ any missing file. but kept as a safety net. - Tools work through the `--jinja` template (`tool_calls` verified). - Images work through the mmproj (verified with a live JPEG). -- One slot: a second concurrent request queues behind the first. +- Four concurrent conversations (one per slot). No per-slot QoS: a long + prefill in one slot slows decodes in the others — see the caveat above. - Harness: `contextWindow: 262144`, `maxTokens: 32768`. ## Troubleshooting -- **`llama_init_from_model: failed to initialize the context: - Gemma4Assistant requires ctx_other to be set` printed at boot** — this - is the `--fit` probe, not a fault. The engine cannot build the assistant - context without the target during measurement, logs the (self-described - normal) error, fits without the extra model, and loads the draft - against the real context seconds later. Do not restart over it. +- **`Gemma4Assistant requires ctx_other to be set` printed at boot** — + the `--fit` probe, not a fault (fit cannot build the assistant context + during measurement). With `CTX_SIZE` set explicitly, as deployed, this + line does not appear at all. Do not restart over it either way. - **Draft looks unengaged**: ngram-mod is silent at load; read the `draft acceptance = …` line per task from `journalctl -u frank-small` (the server must not be restarted just to diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 0952c96..02d1093 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -820,3 +820,31 @@ timestamp: 2026-09-14 set` error is `--fit`'s own probe — normal, self-recovers seconds later. - Runbook: [[frank-small-gemma4-12b]] (active); [[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target. + +## 2026-09-17 (later) — frank-small: four full windows, max-length throughput sweep + +- **Slot ceiling measured:** 1 full-ctx slot idles 11,996 MiB, 4 idle + 22,346 MiB -> 3,450 MiB/slot, ~8,546 fixed; a 5th projects past the + 24,576 MiB card. Deployed `CTX_SIZE=1048576` / `PARALLEL=4` (each + slot the FULL 262,144 window; explicit CTX bypasses `--fit` on + purpose — fit would divide the pool at parallel>1). The bats seam now + pins the division and the budget inequality. +- **Audio verified:** JFK WAV via `input_audio` -> verbatim transcript. +- **Batch sweep at max length** (salted API probes, pp ~254k / + tg @ ~253k ctx): ubatch 512 -> pp 912–924, tg 90–204; + **ubatch 1024 -> pp 962–965, tg 226–302 (deployed)**; + ubatch 2048 -> OOM in the MTP draft's pp compute buffer (~3.3 GiB + vs 2.2 GiB headroom). +- **Negative results kept:** draft n-max 4 + poll 100 decoded WORSE at + 250k (230–257) — longer verification rounds re-read more long-ctx + attention than the accepted token earns. No CUDA-graph flags exist in + this build, so launch-overhead tuning is off the table. +- **Concurrency caveat:** a 254k prefill beside a decode drops it + ~240 -> ~100–135 t/s; one engine, no per-slot QoS. +- **Housekeeping during the sweep:** the earlier live service was + discovered to be an orphaned manual boot holding 7771 past the unit + stop (kill -9'd); the sweep now refuses to measure a port it does not + own. Peer frank-medium WIP was stash-held around the deploy and + self-committed as `23f0978`/`74dda30` — nothing lost. +- **Live after deploy:** `n_slots = 4, n_ctx_slot = 262144`, idle + 23,300 MiB, e2e gate 32 passed.