From 87caca48940e33e75ec5360d7f2bb0d820ca5634 Mon Sep 17 00:00:00 2001 From: Frank Date: Thu, 17 Sep 2026 02:18:44 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20frank-small=20is=20Gemma4-12B=20QAT=20n?= =?UTF-8?q?ow=20=E2=80=94=20new=20runbook,=20Qwen=20retired=20as=20rollbac?= =?UTF-8?q?k=20target?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated. --- AGENTS.md | 44 ++++-- .../LLM/frank-small-gemma4-12b.md | 148 ++++++++++++++++++ .../LLM/frank-small-qwen38-gsq-rco.md | 11 +- serverAppConfigs-docs/LLM/index.md | 12 +- serverAppConfigs-docs/index.md | 13 +- serverAppConfigs-docs/log.md | 27 ++++ 6 files changed, 226 insertions(+), 29 deletions(-) create mode 100644 serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md diff --git a/AGENTS.md b/AGENTS.md index ead7ed7..87868bf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). |--------|---------|---------|-------|------| | Dankserver | `frank-medium` | mainline llama.cpp master fb27a525 (root systemd unit, gfx1201 HIP build) | ajgazin Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2 Q8_0 drafter, 2x Radeon AI PRO R9700 in ONE layer-split process (36:29), 327680 ctx over 2 slots of 163840, batch 6144x512, vision (BF16 mmproj) | 7770 | | Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 | -| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 | +| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | HauhauCS Gemma4-12B-QAT-Uncensored-Balanced Q4_K_M (BF16 mmproj + MTP assistant sidecar, `draft-mtp,ngram-mod`) on the RTX 3090, `--fit` → 262,144 ctx / 1 slot, q8_0 KV, the fleet's vision endpoint | 7771 | Retired endpoints (`frank-npu`, and every prior runtime) are in the [runbook index](#runbook-index). The full swap history lives in @@ -299,20 +299,31 @@ Rollback: [[frank-large-ik-uncensored]]. ### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE) -Mainline llama.cpp master `37b53fd` as a **root systemd unit** (repo +Mainline llama.cpp master `37b53fd` (CUDA build with +`-DGGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit** (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`), -serving `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S — the `-mtp` build, so -the model's own Multi-Token Prediction head drives speculative decoding — with -the BF16 mmproj, on port 7771, 131,072 ctx / 1 slot, f16 KV, greedy default. -Deployed flags are in `scripts/frank-small.env`; the serve script refuses to -start if the weights, the projector or the binary are missing. +serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M. +The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4 +nextn layers) loaded via `--model-draft`; the draftless **ngram-mod** +matcher chains behind it in the same `--spec-type` value. BF16 mmproj, +port 7771, q8_0 KV, `CTX_SIZE` unset so `--fit` sizes the pool, 1 slot, +greedy default. Deployed flags are in `scripts/frank-small.env`; the serve +script refuses to start if the weights, the projector, the sidecar or the +binary are missing. -Verified at deploy (2026-09-16): `/v1/models` reports `frank-small`; a chat -completion returns content plus `reasoning_content`; a tool call returns -`tool_calls`; the slot timing lines show `draft acceptance = 0.66–0.94, mean -len ≈ 3` (MTP engaged); an image request is answered from the projector while -the halogen side refuses images with a named reason; 22,362 MiB of the card's -24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`. +Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a +chat completion returns content plus `reasoning_content`; a tool call +returns `tool_calls`; the slot timing lines show `draft acceptance = +0.38–0.46, mean len ≈ 2.2` (MTP engaged); an image request is answered +from the projector. `--fit` took the FULL window — `n_slots = 1, +n_ctx_slot = 262144` — because Gemma4's sliding-window pattern keeps the +q8 KV pool cheap: 11,996 MiB of the card's 24,576 MiB in use. The +`Gemma4Assistant requires ctx_other to be set` error that prints at boot +is the fit's own probe (the log calls it normal); the sidecar then loads +against the real context. Gemma4's template accepts every +`reasoning_effort` level (all seven probed: 200). +Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`. +Rollback: [[frank-small-qwen38-gsq-rco]]. ### frank-medium — llama.cpp layer split, DFlash2 + ngram-mod (Dankserver, ACTIVE) @@ -594,7 +605,9 @@ linked runbooks. returns `tool_calls`. Check tool support before routing a role at an endpoint. - **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same option and the same ~12x prefill collapse described in the ROCm note above. - f16 KV needs nothing, which is why frank-small runs f16. + f16 KV needs nothing; frank-small ran f16 for that reason until + 2026-09-17, when `build-cuda` was rebuilt with the option ON so the + endpoint could take q8_0 KV. - **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device* configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep @@ -685,7 +698,7 @@ Active: | Service | Runbook | |---------|---------| | frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` | -| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` | +| frank-small (llama.cpp Gemma4-12B QAT, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md` | | frank-medium (llama.cpp + DFlash2, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md` | | Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` | | Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` | @@ -696,6 +709,7 @@ Retired (rollback targets and history): |---------|------|--------|---------| | kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` | | ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` | +| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-small`) | Dankclaw | 2026-09-16 → 09-17 | `frank-small-qwen38-gsq-rco.md` | | agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` | | eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` | | Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` | diff --git a/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md b/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md new file mode 100644 index 0000000..4958b68 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md @@ -0,0 +1,148 @@ +--- +type: reference +title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M +description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves. +tags: + - llm + - frank-small + - dankclaw + - llama.cpp + - rtx3090 + - gemma + - mtp + - vision +timestamp: 2026-09-17 +--- + +# frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT) + +> **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]] +> (one-day rollback target). Same unit, same port, same client contract — +> the model, the drafter shape and the KV type changed. + +## Identity + +| what | value | +|---|---| +| runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) | +| binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` | +| unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) | +| model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M | +| model dir | `/home/dank/models/Gemma4-12B-QAT/` | +| port | 7771 (alias `frank-small`; `frank-large` keeps 7770) | +| window | `--fit on` → **262,144** ctx, 1 slot; q8_0 K and V | +| vision | fleet image path via BF16 mmproj (`input: [text, image]`) | + +## Payload (all sha256-verified against the Hub) + +| file | bytes | sha256 (head…) | +|---|---|---| +| `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` | +| `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` | +| `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` | + +`sha256.txt` beside them on the host carries the full digests. + +## What the preflight found (GGUF headers, read over HTTP Range) + +- **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x + 512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30, + **sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings + (PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers + every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`). + 48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch — + cosmetic only. +- **mmproj**: clip, vision 224/16 patches, **also carries an audio + encoder** (`has_audio_encoder=True`) — vision is the fleet contract, + audio is inert bonus. +- **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure + assistant (no trunk). Its KV **shares** the target's last layers — the + share lambda at `llama-model.cpp:2662` references + `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the + drafter therefore costs no second KV pool. + +## KV arithmetic — why 262k fits + +Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) = +**16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the +sliding layers cache only the window; the full-attention layers cache the +full context. The engine's own `--fit` measured the combination and took +the whole 262,144-token window: total footprint at idle is **11,996 MiB** +of the card's 24,576 MiB, weights included. + +This is the practical payoff of a sliding-window architecture: the naive +"all-global" worst case was 32 GiB at q8/262k — impossible; the real +pattern makes the full window the cheapest possible answer. + +## Spec-decode chain + +`--spec-type draft-mtp,ngram-mod` with `--model-draft +$MODEL_DIR/mtp-gemma-4-12B-it.gguf`: + +- `draft-mtp` is context-backed and reads the assistant sidecar. Two + context-backed drafters cannot share one process, so the chain carries + exactly one. +- `ngram-mod` is draftless and attaches unconditionally, catching + prompt-echo traffic MTP would miss. It prints **no startup banner** — + verify behaviorally via the per-task `draft acceptance` line, not the + load log. +- Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2` + (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a + much smaller model — the point of the swap was the VRAM and the window, + not acceptance). + +## Deployed flags + +Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`. +Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`, +`MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, +`DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, +`SPEC_TYPE=draft-mtp,ngram-mod`, `CTX_SIZE` **unset** (→ `--fit on`), +`PARALLEL=1`, `NGL=999`, `FLASH_ATTN=on`, `--jinja`, `--metrics`, +greedy `--temp 0.0` server default. The serve script refuses to start on +any missing file. + +## Client contract + +- `reasoning_content` carries thinking; `preserve_thinking: true` is sent + by the harness to keep it across turns. +- **Every `reasoning_effort` level is accepted** (all seven probed + 2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's + `reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert + but kept as a safety net. +- Tools work through the `--jinja` template (`tool_calls` verified). +- Images work through the mmproj (verified with a live JPEG). +- One slot: a second concurrent request queues behind the first. +- Harness: `contextWindow: 262144`, `maxTokens: 32768`. + +## Troubleshooting + +- **`llama_init_from_model: failed to initialize the context: + Gemma4Assistant requires ctx_other to be set` printed at boot** — this + is the `--fit` probe, not a fault. The engine cannot build the assistant + context without the target during measurement, logs the (self-described + normal) error, fits without the extra model, and loads the draft + against the real context seconds later. Do not restart over it. +- **Draft looks unengaged**: ngram-mod is silent at load; read the + `draft acceptance = …` line per task from + `journalctl -u frank-small` (the server must not be restarted just to + see this). +- **Quantized KV regression**: if someone rebuilds `build-cuda` without + `GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see + the CUDA lesson in AGENTS.md). Check + `grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild. +- Logs: `journalctl -u frank-small` (root unit, not quadlet). + +## Rollback + +Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/ +`DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see +[[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen +weights were NOT deleted from the host. + +## References + +- `scripts/serve-frank-small.sh` — the actual exec line +- [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback) +- [[halogen-flash-server]] — the sibling endpoint on the iGPU +- [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk diff --git a/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md b/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md index 671a4e4..6911db5 100644 --- a/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md +++ b/serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md @@ -1,7 +1,7 @@ --- type: reference title: frank-small on Dankclaw — llama.cpp, Qwen3.8-27B GSQ-RCO IQ3_S -description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves. +description: The RETIRED frank-small runbook (2026-09-16 → 09-17, rollback target) — mainline llama.cpp on Dankclaw's RTX 3090 serving ISTA-DASLab Qwen3.8-27B-GSQ-RCO at IQ3_S with the model's own in-file MTP head and the BF16 vision projector: identity, KV arithmetic, deployed flags, the MTP contract, the client contract, and what to check when it misbehaves. tags: - llm - frank-small @@ -16,10 +16,11 @@ timestamp: 2026-09-16 # frank-small — llama.cpp on the RTX 3090 -> **ACTIVE since 2026-09-16.** Created when Dankclaw's inference was re-split: -> [[halogen-flash-server]] took the gfx1151 iGPU back for `frank-large`, which -> freed the RTX 3090 for this endpoint. Two endpoints, one host: `frank-large` -> on `:7770`, `frank-small` on `:7771`. +> **RETIRED 2026-09-17** — replaced by [[frank-small-gemma4-12b]]. Kept as +> the rollback target and the record of the deployment it succeeded. +> Originally ACTIVE from 2026-09-16, created when Dankclaw's inference was +> re-split: [[halogen-flash-server]] took the gfx1151 iGPU back for +> `frank-large`, which freed the RTX 3090 for this endpoint. ## Identity diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index c343561..d60baa6 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -87,11 +87,15 @@ frank-small, port 7771] overlay) on the gfx1151 iGPU — two 200k conversations over one 400,384-position pool, deployed flags, memory model, client contract, troubleshooting -- [[frank-small-qwen38-gsq-rco]] — ACTIVE frank-small runbook since 2026-09-16 +- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17 (root systemd unit, port 7771): mainline llama.cpp serving - ISTA-DASLab/Qwen3.8-27B-GSQ-RCO IQ3_S on the RTX 3090, with the model's own - Multi-Token Prediction head for speculative decoding and the BF16 projector - for the fleet's image path + HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX + 3090, with the separate MTP assistant sidecar chained behind ngram-mod, + q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB +- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the + prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab + Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and + f16 KV at 131k - [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index 3396afa..10cf991 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -165,11 +165,14 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[frank-small-qwen38-gsq-rco]] — the ACTIVE frank-small runbook since - 2026-09-16: mainline llama.cpp serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO- - GGUF at IQ3_S (own MTP head + BF16 vision projector) on Dankclaw's - RTX 3090, port 7771 — the 3090 freed when halogen took the iGPU back for - frank-large +- [[frank-small-gemma4-12b]] — the ACTIVE frank-small runbook since + 2026-09-17: mainline llama.cpp serving + HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M + (MTP assistant sidecar + ngram-mod, q8_0 KV, --fit → 262k) on + Dankclaw's RTX 3090, port 7771 +- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): + the prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab + Qwen3.8-27B-GSQ-RCO IQ3_S with own in-file MTP head + BF16 projector - [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) - [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index b2a5bfb..0952c96 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -793,3 +793,30 @@ timestamp: 2026-09-14 contiguous 2 MiB blocks left (240 before the refresh). - Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the quant comparison is appended to [[x4-pair-experiment]]. + +## 2026-09-17 — frank-small swapped to Gemma4-12B QAT (Q4_K_M) + +- **Cutover:** `frank-small` (Dankclaw RTX 3090, :7771) replaced + ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with + `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M — + 7.38 GiB main + 175 MiB BF16 mmproj (with audio encoder) + 254 MiB + `mtp-gemma-4-12B-it.gguf` assistant sidecar, all sha256-verified. +- **Spec decode:** the MTP head is now a separate `gemma4-assistant` + file loaded via `--model-draft` (shares the target's last layers' KV, + `llama-model.cpp:2662`), chained behind draftless `ngram-mod`: + `--spec-type draft-mtp,ngram-mod`. Measured acceptance 0.38–0.46, + mean draft len ~2.2. +- **KV:** q8_0 K and V; `build-cuda` rebuilt with + `-DGGML_CUDA_FA_ALL_QUANTS=ON` (required, else ~12x prefill collapse). + `CTX_SIZE` unset → `--fit on` chose the full 262,144-token window; + idle card usage 11,996 / 24,576 MiB (was 22,362). +- **Template behavior:** Gemma4 accepts every `reasoning_effort` level + (all seven probed: 200) — the Qwen3.8 gate is gone; harness map kept + as a safety net. +- **Verified live:** health 200, chat + `reasoning_content`, `tool_calls`, + vision (live JPEG), deploy e2e gate passed (31 passed, 1 flaky crawl). +- **Harness:** `models.yml` frank-small `contextWindow: 262144`. +- **Note:** the boot log's `Gemma4Assistant requires ctx_other to be + set` error is `--fit`'s own probe — normal, self-recovers seconds later. +- Runbook: [[frank-small-gemma4-12b]] (active); + [[frank-small-qwen38-gsq-rco]] retired 2026-09-17 as rollback target.