serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md

---

type: reference title: frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M description: The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves. tags:

• llm
• frank-small
• dankclaw
• llama.cpp
• rtx3090
• gemma
• mtp
• vision timestamp: 2026-09-17

---

frank-small — llama.cpp on the RTX 3090 (Gemma4-12B QAT)

ACTIVE since 2026-09-17. Replaces frank-small-qwen38-gsq-rco (one-day rollback target). Same unit, same port, same client contract — the model, the drafter shape and the KV type changed.

Identity

what	value
runtime	mainline llama.cpp master 37b53fd, CUDA build with -DGGML_CUDA_FA_ALL_QUANTS=ON (rebuilt 2026-09-17 for q8 KV)
binary	/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
unit	root systemd unit frank-small.service (repo scripts/frank-small.service, relink_united into /etc/systemd/system/)
model	HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M
model dir	/home/dank/models/Gemma4-12B-QAT/
port	7771 (alias frank-small; frank-large keeps 7770)
window	--fit on → 262,144 ctx, 1 slot; q8_0 K and V
vision	fleet image path via BF16 mmproj (input: [text, image])

Payload (all sha256-verified against the Hub)

file	bytes	sha256 (head…)
Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf	7,381,381,760	59656d74…
mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf	175,115,264	b59e8154…
mtp-gemma-4-12B-it.gguf	253,707,328	c50c91c3…

sha256.txt beside them on the host carries the full digests.

What the preflight found (GGUF headers, read over HTTP Range)

• Target (arch gemma4, 48 blocks): ctx 262144, MHA — 16 kv heads x 512 dim (no GQA), embedding 3840, rope base 1e6, logit softcap 30, sliding_window 1024, swa_type = STANDARD, per-layer token embeddings (PLE) via LLM_KV_EMBEDDING_LENGTH_PER_LAYER. 12 full-attention layers every 4th ([3,7,…,47] implied by full_attention_interval = 4). 48 blocks maps to LLM_TYPE_UNKNOWN in the loader's size switch — cosmetic only.
• mmproj: clip, vision 224/16 patches, also carries an audio encoder (has_audio_encoder=True) — vision is the fleet contract, audio is inert bonus.
• MTP sidecar (arch gemma4-assistant): 4 nextn layers, pure assistant (no trunk). Its KV shares the target's last layers — the share lambda at llama-model.cpp:2662 references llama_model_n_layer(model_other) - 2 for SWA layers. Adding the drafter therefore costs no second KV pool.

KV arithmetic — why 262k fits

Per token per layer at q8_0: 16 heads x 512 dim x 1 B x 2 (K,V) = 16 KiB (32 KiB at f16). With SWA_STANDARD and window 1024, the sliding layers cache only the window; the full-attention layers cache the full context. The engine's own --fit measured the combination and took the whole 262,144-token window: total footprint at idle is 11,996 MiB of the card's 24,576 MiB, weights included.

This is the practical payoff of a sliding-window architecture: the naive "all-global" worst case was 32 GiB at q8/262k — impossible; the real pattern makes the full window the cheapest possible answer.

Spec-decode chain

--spec-type draft-mtp,ngram-mod with --model-draft $MODEL_DIR/mtp-gemma-4-12B-it.gguf:

• draft-mtp is context-backed and reads the assistant sidecar. Two context-backed drafters cannot share one process, so the chain carries exactly one.
• ngram-mod is draftless and attaches unconditionally, catching prompt-echo traffic MTP would miss. It prints no startup banner — verify behaviorally via the per-task draft acceptance line, not the load log.
• Measured at deploy: draft acceptance = 0.38–0.46, mean len ≈ 2.2 (lower acceptance than the retired Qwen in-file MTP's 0.66–0.94, but a much smaller model — the point of the swap was the VRAM and the window, not acceptance).

Deployed flags

Repo truth: scripts/frank-small.env + scripts/serve-frank-small.sh. Key values: MODEL_DIR=/home/dank/models/Gemma4-12B-QAT, MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf, DRAFT_FILE=mtp-gemma-4-12B-it.gguf, CACHE_TYPE_K=V=q8_0, SPEC_TYPE=draft-mtp,ngram-mod, CTX_SIZE unset (→ --fit on), PARALLEL=1, NGL=999, FLASH_ATTN=on, --jinja, --metrics, greedy --temp 0.0 server default. The serve script refuses to start on any missing file.

Client contract

• reasoning_content carries thinking; preserve_thinking: true is sent by the harness to keep it across turns.
• Every reasoning_effort level is accepted (all seven probed 2026-09-17: 200) — unlike Qwen3.8's template gate. The harness's reasoningEffortMap in ~/.omp/agent/models.yml is therefore inert but kept as a safety net.
• Tools work through the --jinja template (tool_calls verified).
• Images work through the mmproj (verified with a live JPEG).
• One slot: a second concurrent request queues behind the first.
• Harness: contextWindow: 262144, maxTokens: 32768.

Troubleshooting

• llama_init_from_model: failed to initialize the context: Gemma4Assistant requires ctx_other to be set printed at boot — this is the --fit probe, not a fault. The engine cannot build the assistant context without the target during measurement, logs the (self-described normal) error, fits without the extra model, and loads the draft against the real context seconds later. Do not restart over it.
• Draft looks unengaged: ngram-mod is silent at load; read the draft acceptance = … line per task from journalctl -u frank-small (the server must not be restarted just to see this).
• Quantized KV regression: if someone rebuilds build-cuda without GGML_CUDA_FA_ALL_QUANTS=ON, prefill collapses ~12x with q8 KV (see the CUDA lesson in AGENTS.md). Check grep FA_ALL_QUANTS build-cuda/CMakeCache.txt after any rebuild.
• Logs: journalctl -u frank-small (root unit, not quadlet).

Rollback

Redeploying the prior model means repointing MODEL_DIR/MODEL_FILE/ DRAFT_FILE back to the Qwen trio and SPEC_TYPE=draft-mtp — see frank-small-qwen38-gsq-rco for the exact deployed values. The Qwen weights were NOT deleted from the host.

References

• scripts/serve-frank-small.sh — the actual exec line
• frank-small-qwen38-gsq-rco — RETIRED predecessor (rollback)
• halogen-flash-server — the sibling endpoint on the iGPU
• x4-pair-experiment — why the 3090 serves its own model, not the trunk