serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md

---

type: reference title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves. tags:

• llm
• frank-small
• dankclaw
• llama.cpp
• rtx3090
• gemma
• mtp
• vision timestamp: 2026-09-17

---

frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)

ACTIVE since 2026-09-17. Replaces frank-small-qwen38-gsq-rco (one-day rollback target). Same unit, same port, same client contract — the model, the drafter shape and the KV type changed.

Identity

what	value
runtime	mainline llama.cpp master 37b53fd, CUDA build with -DGGML_CUDA_FA_ALL_QUANTS=ON (rebuilt 2026-09-17 for q8 KV)
binary	/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
unit	root systemd unit frank-small.service (repo scripts/frank-small.service, relink_united into /etc/systemd/system/)
model	HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M
model dir	/home/dank/models/Gemma4-12B-QAT/
port	7771 (alias frank-small; frank-large keeps 7770)
window	4 slots x 262,144 (CTX_SIZE=1048576, PARALLEL=4); q8_0 K and V
vision + audio	fleet image path via BF16 mmproj (input: [text, image]); input_audio transcribes verbatim (verified)

Payload (all sha256-verified against the Hub)

file	bytes	sha256 (head…)
Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf	7,381,381,760	59656d74…
mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf	175,115,264	b59e8154…
mtp-gemma-4-12B-it.gguf	253,707,328	c50c91c3…

sha256.txt beside them on the host carries the full digests.

What the preflight found (GGUF headers, read over HTTP Range)

• Target (arch gemma4, 48 blocks): ctx 262144, MHA — 16 kv heads x 512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30, sliding_window 1024, swa_type = STANDARD, per-layer token embeddings (PLE) via LLM_KV_EMBEDDING_LENGTH_PER_LAYER. 12 full-attention layers every 4th ([3,7,…,47] implied by full_attention_interval = 4). 48 blocks maps to LLM_TYPE_UNKNOWN in the loader's size switch — cosmetic only.
• mmproj: clip, vision 224/16 patches, also carries an audio encoder (has_audio_encoder=True) — vision is the fleet contract, and audio works: a JFK WAV sent as an input_audio content part was transcribed verbatim (2026-09-17).
• MTP sidecar (arch gemma4-assistant): 4 nextn layers, pure assistant (no trunk). Its KV shares the target's last layers — the share lambda at llama-model.cpp:2662 references llama_model_n_layer(model_other) - 2 for SWA layers. Adding the drafter therefore costs no second KV pool.

KV arithmetic — why four full windows fit

Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) = 16 KiB (32 KiB at f16). With SWA_STANDARD and window 1024, the sliding layers cache only the window; the full-attention layers cache the full context — the pattern that makes the window cheap (the naive "all-global" worst case was 32 GiB at q8/262k, impossible).

Measured on the 3090 (24,576 MiB): one full-context slot idles at 11,996 MiB, four at 22,346 MiB — 3,450 MiB per slot, ~8,546 MiB fixed. A fifth slot projects past the card, so 4 x 262,144 is the ceiling. CTX_SIZE is set explicitly to PARALLEL x 262144 rather than left for --fit: fit maximises the TOTAL pool and would divide a smaller window per slot at parallel > 1. The slot-grid seam in tests/frank_small_unit.bats pins both the full-window division and the budget inequality so an oversized bump fails pre-deploy, not in boot.

Spec-decode chain

--spec-type draft-mtp,ngram-mod with --model-draft $MODEL_DIR/mtp-gemma-4-12B-it.gguf:

• draft-mtp is context-backed and reads the assistant sidecar. Two context-backed drafters cannot share one process, so the chain carries exactly one.
• ngram-mod is draftless and attaches unconditionally, catching prompt-echo traffic MTP would miss. It prints no startup banner — verify behaviorally via the per-task draft acceptance line, not the load log.
• Measured at deploy: draft acceptance = 0.38–0.46, mean len ≈ 2.2 (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a much smaller model — the point of the swap was the VRAM and the window, not acceptance).

Throughput tuning (measured 2026-09-17)

Salted API probes at the deployed geometry (4 slots, q8 KV, flash attn), prefill ~254k tokens / decode at ~253k context:

batch (b/ub)	draft n-max / poll	pp t/s	tg@250k t/s	idle MiB
2048/512 (old)	3 / 50	912–924	90*–204	22,346
8192/1024 (deployed)	3 / 50	962–965	226–302	23,300
8192/1024	4 / 100	961	230–257	23,300
4096/1024	4 / 100	959	205–277	23,300
8192/2048	—	—	—	OOM

• ubatch 1024 is the winner: +5% max-length prefill, and decode at 250k clearly above the 512 baseline.
• ubatch 2048 cannot run: the MTP draft context needs ~3.3 GiB of pp compute buffer at that ubatch against ~2.2 GiB headroom — it dies in common_speculative_init_result after the main model loads.
• Deeper draft and busy-wait HURT long-ctx decode: draft n-max 4 with poll 100 verified a longer round per token, and at 250k the extra full-attention re-read outweighs the accepted-token gain (230–257 vs 257–302). Keep 3 / 50.
• All ubatch-1024 variants idle identically (23,300 MiB) — compute buffers are reserved at max graph size regardless of the logical batch, so 8192 costs nothing over 4096.
• Concurrency caveat: a long prefill monopolises the GPU — a decode running beside a 254k-token prefill drops from ~240 to ~100–135 t/s. Four slots share one engine with no per-slot QoS; that is inherent, not a config fault.
• No CUDA graphs in this build — --help shows zero graph flags; the decode-launch-overhead lever does not exist here.

Deployed flags

Repo truth: scripts/frank-small.env + scripts/serve-frank-small.sh. Key values: MODEL_DIR=/home/dank/models/Gemma4-12B-QAT, MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf, DRAFT_FILE=mtp-gemma-4-12B-it.gguf, CACHE_TYPE_K=V=q8_0, SPEC_TYPE=draft-mtp,ngram-mod, SPEC_DRAFT_N_MAX=3, POLL_LEVEL=50, CTX_SIZE=1048576 with PARALLEL=4 (four full 262,144-token slots), BATCH_SIZE=8192, UBATCH_SIZE=1024, NGL=999, FLASH_ATTN=on, --jinja, --metrics, greedy --temp 0.0 server default. The serve script refuses to start on any missing file.

Client contract

• reasoning_content carries thinking; preserve_thinking: true is sent by the harness to keep it across turns.
• Every reasoning_effort level is accepted (all seven probed 2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's reasoningEffortMap in ~/.omp/agent/models.yml is therefore inert but kept as a safety net.
• Tools work through the --jinja template (tool_calls verified).
• Images work through the mmproj (verified with a live JPEG).
• Four concurrent conversations (one per slot). No per-slot QoS: a long prefill in one slot slows decodes in the others — see the caveat above.
• Harness: contextWindow: 262144, maxTokens: 32768.

Troubleshooting

• Gemma4Assistant requires ctx_other to be set printed at boot — the --fit probe, not a fault (fit cannot build the assistant context during measurement). With CTX_SIZE set explicitly, as deployed, this line does not appear at all. Do not restart over it either way.
• Draft looks unengaged: ngram-mod is silent at load; read the draft acceptance = … line per task from journalctl -u frank-small (the server must not be restarted just to see this).
• Quantized KV regression: if someone rebuilds build-cuda without GGML_CUDA_FA_ALL_QUANTS=ON, prefill collapses ~12x with q8 KV (see the CUDA lesson in AGENTS.md). Check grep FA_ALL_QUANTS build-cuda/CMakeCache.txt after any rebuild.
• Logs: journalctl -u frank-small (root unit, not quadlet).

Rollback

Redeploying the prior model means repointing MODEL_DIR/MODEL_FILE/ DRAFT_FILE back to the Qwen trio and SPEC_TYPE=draft-mtp — see frank-small-qwen38-gsq-rco for the exact deployed values. The Qwen weights were NOT deleted from the host.

References

• scripts/serve-frank-small.sh — the actual exec line
• frank-small-qwen38-gsq-rco — RETIRED predecessor (rollback)
• halogen-flash-server — the sibling endpoint on the iGPU
• x4-pair-experiment — why the 3090 serves its own model, not the trunk