--- type: reference title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves. tags: - llm - frank-small - dankclaw - llama.cpp - rtx3090 - gemma - mtp - vision timestamp: 2026-09-17 --- # frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT) > **ACTIVE since 2026-09-17.** Replaces [[frank-small-qwen38-gsq-rco]] > (one-day rollback target). Same unit, same port, same client contract — > the model, the drafter shape and the KV type changed. ## Identity | what | value | |---|---| | runtime | mainline llama.cpp master `37b53fd`, CUDA build **with `-DGGML_CUDA_FA_ALL_QUANTS=ON`** (rebuilt 2026-09-17 for q8 KV) | | binary | `/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server` | | unit | **root systemd unit** `frank-small.service` (repo `scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`) | | model | `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M | | model dir | `/home/dank/models/Gemma4-12B-QAT/` | | port | 7771 (alias `frank-small`; `frank-large` keeps 7770) | | window | `--fit on` → **262,144** ctx, 1 slot; q8_0 K and V | | vision | fleet image path via BF16 mmproj (`input: [text, image]`) | ## Payload (all sha256-verified against the Hub) | file | bytes | sha256 (head…) | |---|---|---| | `Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf` | 7,381,381,760 | `59656d74…` | | `mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf` | 175,115,264 | `b59e8154…` | | `mtp-gemma-4-12B-it.gguf` | 253,707,328 | `c50c91c3…` | `sha256.txt` beside them on the host carries the full digests. ## What the preflight found (GGUF headers, read over HTTP Range) - **Target** (`arch gemma4`, 48 blocks): ctx 262144, MHA — 16 kv heads x 512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30, **sliding_window 1024**, `swa_type = STANDARD`, per-layer token embeddings (PLE) via `LLM_KV_EMBEDDING_LENGTH_PER_LAYER`. 12 full-attention layers every 4th (`[3,7,…,47]` implied by `full_attention_interval = 4`). 48 blocks maps to `LLM_TYPE_UNKNOWN` in the loader's size switch — cosmetic only. - **mmproj**: clip, vision 224/16 patches, **also carries an audio encoder** (`has_audio_encoder=True`) — vision is the fleet contract, audio is inert bonus. - **MTP sidecar** (`arch gemma4-assistant`): 4 nextn layers, pure assistant (no trunk). Its KV **shares** the target's last layers — the share lambda at `llama-model.cpp:2662` references `llama_model_n_layer(model_other) - 2` for SWA layers. Adding the drafter therefore costs no second KV pool. ## KV arithmetic — why 262k fits Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) = **16 KiB** (32 KiB at f16). With SWA_STANDARD and window 1024, the sliding layers cache only the window; the full-attention layers cache the full context. The engine's own `--fit` measured the combination and took the whole 262,144-token window: total footprint at idle is **11,996 MiB** of the card's 24,576 MiB, weights included. This is the practical payoff of a sliding-window architecture: the naive "all-global" worst case was 32 GiB at q8/262k — impossible; the real pattern makes the full window the cheapest possible answer. ## Spec-decode chain `--spec-type draft-mtp,ngram-mod` with `--model-draft $MODEL_DIR/mtp-gemma-4-12B-it.gguf`: - `draft-mtp` is context-backed and reads the assistant sidecar. Two context-backed drafters cannot share one process, so the chain carries exactly one. - `ngram-mod` is draftless and attaches unconditionally, catching prompt-echo traffic MTP would miss. It prints **no startup banner** — verify behaviorally via the per-task `draft acceptance` line, not the load log. - Measured at deploy: `draft acceptance = 0.38–0.46, mean len ≈ 2.2` (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a much smaller model — the point of the swap was the VRAM and the window, not acceptance). ## Deployed flags Repo truth: `scripts/frank-small.env` + `scripts/serve-frank-small.sh`. Key values: `MODEL_DIR=/home/dank/models/Gemma4-12B-QAT`, `MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf`, `DRAFT_FILE=mtp-gemma-4-12B-it.gguf`, `CACHE_TYPE_K=V=q8_0`, `SPEC_TYPE=draft-mtp,ngram-mod`, `CTX_SIZE` **unset** (→ `--fit on`), `PARALLEL=1`, `NGL=999`, `FLASH_ATTN=on`, `--jinja`, `--metrics`, greedy `--temp 0.0` server default. The serve script refuses to start on any missing file. ## Client contract - `reasoning_content` carries thinking; `preserve_thinking: true` is sent by the harness to keep it across turns. - **Every `reasoning_effort` level is accepted** (all seven probed 2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's `reasoningEffortMap` in `~/.omp/agent/models.yml` is therefore inert but kept as a safety net. - Tools work through the `--jinja` template (`tool_calls` verified). - Images work through the mmproj (verified with a live JPEG). - One slot: a second concurrent request queues behind the first. - Harness: `contextWindow: 262144`, `maxTokens: 32768`. ## Troubleshooting - **`llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set` printed at boot** — this is the `--fit` probe, not a fault. The engine cannot build the assistant context without the target during measurement, logs the (self-described normal) error, fits without the extra model, and loads the draft against the real context seconds later. Do not restart over it. - **Draft looks unengaged**: ngram-mod is silent at load; read the `draft acceptance = …` line per task from `journalctl -u frank-small` (the server must not be restarted just to see this). - **Quantized KV regression**: if someone rebuilds `build-cuda` without `GGML_CUDA_FA_ALL_QUANTS=ON`, prefill collapses ~12x with q8 KV (see the CUDA lesson in AGENTS.md). Check `grep FA_ALL_QUANTS build-cuda/CMakeCache.txt` after any rebuild. - Logs: `journalctl -u frank-small` (root unit, not quadlet). ## Rollback Redeploying the prior model means repointing `MODEL_DIR`/`MODEL_FILE`/ `DRAFT_FILE` back to the Qwen trio and `SPEC_TYPE=draft-mtp` — see [[frank-small-qwen38-gsq-rco]] for the exact deployed values. The Qwen weights were NOT deleted from the host. ## References - `scripts/serve-frank-small.sh` — the actual exec line - [[frank-small-qwen38-gsq-rco]] — RETIRED predecessor (rollback) - [[halogen-flash-server]] — the sibling endpoint on the iGPU - [[x4-pair-experiment]] — why the 3090 serves its own model, not the trunk