docs: the max-length sweep tabled; frank-small.env NGL/FLASH_ATTN restored

- runbook: identity now 4x262144, audio verified, throughput table with
  the OOM and negative-result rows, deployed flags updated to reality.
- AGENTS.md frank-small paragraph matches the deployed shape.
- log.md: the sweep entry.
- scripts/frank-small.env: my earlier line-range edit had clobbered
  NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The
  live service was unaffected because the script's defaults match the
  declared values exactly, but the file contract is restored.
Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.

AGENTS.md

@@ -306,22 +306,20 @@ serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
 The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
 nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
 matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
-port 7771, q8_0 KV, `CTX_SIZE` unset so `--fit` sizes the pool, 1 slot,
+port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
-greedy default. Deployed flags are in `scripts/frank-small.env`; the serve
+(four FULL 262,144-token slots — the measured card ceiling), batch
-script refuses to start if the weights, the projector, the sidecar or the
+**8192/1024**, greedy default. Deployed flags are in
-binary are missing.
+`scripts/frank-small.env`; the serve script refuses to start if the
+weights, the projector, the sidecar or the binary are missing.
 
 Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
 chat completion returns content plus `reasoning_content`; a tool call
-returns `tool_calls`; the slot timing lines show `draft acceptance =
+returns `tool_calls`; an image request is answered from the projector;
-0.38–0.46, mean len ≈ 2.2` (MTP engaged); an image request is answered
+an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
-from the projector. `--fit` took the FULL window — `n_slots = 1,
+max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
-n_ctx_slot = 262144` — because Gemma4's sliding-window pattern keeps the
+acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
-q8 KV pool cheap: 11,996 MiB of the card's 24,576 MiB in use. The
+ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
-`Gemma4Assistant requires ctx_other to be set` error that prints at boot
+100 is WORSE at long ctx) is tabled in the runbook.
-is the fit's own probe (the log calls it normal); the sidecar then loads
-against the real context. Gemma4's template accepts every
-`reasoning_effort` level (all seven probed: 200).
 Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
 Rollback: [[frank-small-qwen38-gsq-rco]].
 

scripts/frank-small.env

@@ -89,6 +89,11 @@ UBATCH_SIZE=1024
 # -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
 NUM_THREADS=8
 THREAD_BATCH=8
+
+# Every layer on the 3090, with flash attention (required for q8 KV).
+NGL=999
+FLASH_ATTN=on
+
 # Speculative decoding: the MTP sidecar (draft context) chained with the
 # draftless ngram-mod matcher. Draft depth and busy-wait level are made
 # explicit here (both were the implicit build defaults: 3 and 50).

serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md

@@ -30,8 +30,8 @@ timestamp: 2026-09-17
 | model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M |
 | model dir | `/home/dank/models/Gemma4-12B-QAT/` |
 | port | 7771 (alias `frank-small`; `frank-large` keeps 7770) |
-| window | `--fit on` → **262,144** ctx, 1 slot; q8_0 K and V |
+| window | **4 slots x 262,144** (`CTX_SIZE=1048576`, `PARALLEL=4`); q8_0 K and V |
-| vision | fleet image path via BF16 mmproj (`input: [text, image]`) |
+| vision + audio | fleet image path via BF16 mmproj (`input: [text, image]`); `input_audio` transcribes verbatim (verified) |
 
 ## Payload (all sha256-verified against the Hub)
 
@@ -54,25 +54,30 @@ timestamp: 2026-09-17
   cosmetic only.
 - **mmproj**: clip, vision 224/16 patches, **also carries an audio
   encoder** (`has_audio_encoder=True`) — vision is the fleet contract,
-  audio is inert bonus.
+  and audio works: a JFK WAV sent as an `input_audio` content part was
+  transcribed verbatim (2026-09-17).
 - **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure
   assistant (no trunk). Its KV **shares** the target's last layers — the
   share lambda at `llama-model.cpp:2662` references
   `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the
   drafter therefore costs no second KV pool.
 
-## KV arithmetic — why 262k fits
+## KV arithmetic — why four full windows fit
 
 Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) =
 **16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the
 sliding layers cache only the window; the full-attention layers cache the
-full context. The engine's own `--fit` measured the combination and took
+full context — the pattern that makes the window cheap (the naive
-the whole 262,144-token window: total footprint at idle is **11,996 MiB**
+"all-global" worst case was 32 GiB at q8/262k, impossible).
-of the card's 24,576 MiB, weights included.
 
-This is the practical payoff of a sliding-window architecture: the naive
+Measured on the 3090 (24,576 MiB): one full-context slot idles at
-"all-global" worst case was 32 GiB at q8/262k — impossible; the real
+11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed.
-pattern makes the full window the cheapest possible answer.
+A fifth slot projects past the card, so **4 x 262,144 is the ceiling**.
+`CTX_SIZE` is set explicitly to `PARALLEL x 262144` rather than left for
+`--fit`: fit maximises the TOTAL pool and would divide a smaller window
+per slot at `parallel > 1`. The slot-grid seam in
+`tests/frank_small_unit.bats` pins both the full-window division and the
+budget inequality so an oversized bump fails pre-deploy, not in boot.
 
 ## Spec-decode chain
 
@@ -91,16 +96,49 @@ $MODEL_DIR/mtp-gemma-4-12B-it.gguf`:
   much smaller model — the point of the swap was the VRAM and the window,
   not acceptance).
 
+## Throughput tuning (measured 2026-09-17)
+
+Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn),
+prefill ~254k tokens / decode at ~253k context:
+
+| batch (b/ub) | draft n-max / poll | pp t/s | tg@250k t/s | idle MiB |
+|---|---|---|---|---|
+| 2048/512 (old) | 3 / 50 | 912–924 | 90*–204 | 22,346 |
+| **8192/1024 (deployed)** | **3 / 50** | **962–965** | **226–302** | **23,300** |
+| 8192/1024 | 4 / 100 | 961 | 230–257 | 23,300 |
+| 4096/1024 | 4 / 100 | 959 | 205–277 | 23,300 |
+| 8192/2048 | — | — | — | OOM |
+
+- **ubatch 1024 is the winner**: +5% max-length prefill, and decode at
+  250k clearly above the 512 baseline.
+- **ubatch 2048 cannot run**: the MTP draft context needs ~3.3 GiB of pp
+  compute buffer at that ubatch against ~2.2 GiB headroom — it dies in
+  `common_speculative_init_result` after the main model loads.
+- **Deeper draft and busy-wait HURT long-ctx decode**: draft n-max 4 with
+  poll 100 verified a longer round per token, and at 250k the extra
+  full-attention re-read outweighs the accepted-token gain (230–257 vs
+  257–302). Keep 3 / 50.
+- **All ubatch-1024 variants idle identically** (23,300 MiB) — compute
+  buffers are reserved at max graph size regardless of the logical batch,
+  so 8192 costs nothing over 4096.
+- **Concurrency caveat**: a long prefill monopolises the GPU — a decode
+  running beside a 254k-token prefill drops from ~240 to ~100–135 t/s.
+  Four slots share one engine with no per-slot QoS; that is inherent, not
+  a config fault.
+- **No CUDA graphs in this build** — `--help` shows zero graph flags; the
+  decode-launch-overhead lever does not exist here.
+
 ## Deployed flags
 
 Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`.
 Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`,
 `MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`,
 `DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`,
-`SPEC_TYPE=draft-mtp,ngram-mod`, `CTX_SIZE` **unset** (→ `--fit on`),
+`SPEC_TYPE=draft-mtp,ngram-mod`, `SPEC_DRAFT_N_MAX=3`, `POLL_LEVEL=50`,
-`PARALLEL=1`, `NGL=999`, `FLASH_ATTN=on`, `--jinja`, `--metrics`,
+`CTX_SIZE=1048576` with `PARALLEL=4` (four full 262,144-token slots),
-greedy `--temp 0.0` server default. The serve script refuses to start on
+`BATCH_SIZE=8192`, `UBATCH_SIZE=1024`, `NGL=999`, `FLASH_ATTN=on`,
-any missing file.
+`--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve
+script refuses to start on any missing file.
 
 ## Client contract
 
@@ -112,17 +150,16 @@ any missing file.
   but kept as a safety net.
 - Tools work through the `--jinja` template (`tool_calls` verified).
 - Images work through the mmproj (verified with a live JPEG).
-- One slot: a second concurrent request queues behind the first.
+- Four concurrent conversations (one per slot). No per-slot QoS: a long
+  prefill in one slot slows decodes in the others — see the caveat above.
 - Harness: `contextWindow: 262144`, `maxTokens: 32768`.
 
 ## Troubleshooting
 
-- **`llama_init_from_model: failed to initialize the context:
+- **`Gemma4Assistant requires ctx_other to be set` printed at boot** —
-  Gemma4Assistant requires ctx_other to be set` printed at boot** — this
+  the `--fit` probe, not a fault (fit cannot build the assistant context
-  is the `--fit` probe, not a fault. The engine cannot build the assistant
+  during measurement). With `CTX_SIZE` set explicitly, as deployed, this
-  context without the target during measurement, logs the (self-described
+  line does not appear at all. Do not restart over it either way.
-  normal) error, fits without the extra model, and loads the draft
-  against the real context seconds later. Do not restart over it.
 - **Draft looks unengaged**: ngram-mod is silent at load; read the
   `draft acceptance = …` line per task from
   `journalctl -u frank-small` (the server must not be restarted just to

serverAppConfigs-docs/log.md

@@ -820,3 +820,31 @@ timestamp: 2026-09-14
   set` error is `--fit`'s own probe — normal, self-recovers seconds later.
 - Runbook: [[frank-small-gemma4-12b]] (active);
   [[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target.
+
+## 2026-09-17 (later) — frank-small: four full windows, max-length throughput sweep
+
+- **Slot ceiling measured:** 1 full-ctx slot idles 11,996 MiB, 4 idle
+  22,346 MiB -> 3,450 MiB/slot, ~8,546 fixed; a 5th projects past the
+  24,576 MiB card. Deployed `CTX_SIZE=1048576` / `PARALLEL=4` (each
+  slot the FULL 262,144 window; explicit CTX bypasses `--fit` on
+  purpose — fit would divide the pool at parallel>1). The bats seam now
+  pins the division and the budget inequality.
+- **Audio verified:** JFK WAV via `input_audio` -> verbatim transcript.
+- **Batch sweep at max length** (salted API probes, pp ~254k /
+  tg @ ~253k ctx): ubatch 512 -> pp 912–924, tg 90–204;
+  **ubatch 1024 -> pp 962–965, tg 226–302 (deployed)**;
+  ubatch 2048 -> OOM in the MTP draft's pp compute buffer (~3.3 GiB
+  vs 2.2 GiB headroom).
+- **Negative results kept:** draft n-max 4 + poll 100 decoded WORSE at
+  250k (230–257) — longer verification rounds re-read more long-ctx
+  attention than the accepted token earns. No CUDA-graph flags exist in
+  this build, so launch-overhead tuning is off the table.
+- **Concurrency caveat:** a 254k prefill beside a decode drops it
+  ~240 -> ~100–135 t/s; one engine, no per-slot QoS.
+- **Housekeeping during the sweep:** the earlier live service was
+  discovered to be an orphaned manual boot holding 7771 past the unit
+  stop (kill -9'd); the sweep now refuses to measure a port it does not
+  own. Peer frank-medium WIP was stash-held around the deploy and
+  self-committed as `23f0978`/`74dda30` — nothing lost.
+- **Live after deploy:** `n_slots = 4, n_ctx_slot = 262144`, idle
+  23,300 MiB, e2e gate 32 passed.