serverAppConfigs/serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	frank-small on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M	The active frank-small runbook — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.	llm frank-small dankclaw llama.cpp rtx3090 gemma mtp vision	2026-09-17

frank-small — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

ACTIVE since 2026-09-17. Replaces frank-small-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and frank-small-vllm-gemma4 (the same-day vLLM experiment, also the rollback target). Same port (7771), same alias (frank-small), same client contract surface.

Identity

• Endpoint: frank-small at 192.168.0.143:7771 (Dankclaw).
• Runtime: mainline llama.cpp master 37b53fd, CUDA build for sm_86 with -DGGML_CUDA_FA_ALL_QUANTS=ON (/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server). The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention collapses prefill ~12x on CUDA.
• Service shape: root systemd unit scripts/frank-small.service (User=dank), launched by scripts/serve-frank-small.sh with tunables from scripts/frank-small.env.

Shape

• Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (6.87 GiB). Architecture gemma4: 48 layers, MHA with 16 kv heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer token embeddings (PLE) inside the file, and a sliding-window attention pattern (attention.sliding_window = 1024) — most layers cache only a 1024-position window, which is why the KV pool is a fraction of the full-ctx x all-layers number.
• Context: four FULL 262,144-token slots (CTX_SIZE=1048576, PARALLEL=4). The explicit CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, which would shrink each slot below the full window once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past the card. 4 is the measured max.
• Placement: the whole model in VRAM (--gpu-layers 999); the host CPU only tokenizes and samples (its CPU belongs to the halogen engine serving frank-large).

Spec method

The MTP head ships as a separate file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 GiB), loaded via --model-draft with --spec-type draft-mtp. The same --spec-type value chains the draftless ngram-mod matcher, which attaches unconditionally; two context-backed drafters cannot share one process, so draft-mtp,ngram-mod is the only legal shape that carries both. Draft depth 3 + poll 50 (measured: depth 4 + poll 100 decoded WORSE at long context — the longer verification round costs more attention re-read than the extra accepted token earns).

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), with flash attention (--flash-attn on, required for quantized KV). CPU-side cache capped at 2048 MiB (CACHE_RAM=2048) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the 3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so the >12.8 GiB floor does not apply.

Vision + audio

--mmproj loads the BF16 projector (mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf, 11 tensors, 52M params), which also carries an audio encoder. The fleet's image path lives here. Audio works: input_audio content parts transcribe verbatim (verified 2026-09-17 with a 16k mono WAV).

Payload

The main GGUF, the BF16 mmproj, and the MTP sidecar live under /home/dank/models/Gemma4-12B-QAT. The serve script refuses to start if any is missing (a missing file is an unfinished download, and the unit's restart policy would otherwise turn it into a crash loop that reads like a runtime fault).

Client contract

• Alias: frank-small (the id at /v1/models).
• Reasoning: extracted to reasoning_content (--reasoning-format deepseek, unlimited budget).
• Tools: supported (--jinja is pinned even though this build defaults it on, so a future build that flips the default cannot silently take tools away).
• Vision + audio: supported.
• Sampling: greedy by default (fleet contract); a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Measured (max length)

At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024: pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches (2048) OOM the MTP draft context's pp compute buffer and must not be tried.

Rollback

Payload and runtime for frank-small-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. frank-small-vllm-gemma4 is the same-day vLLM experiment (also a rollback target).

Citations

• Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
• llama.cpp: https://github.com/ggml-org/llama.cpp
• Serve script: scripts/serve-frank-small.sh; tunables: scripts/frank-small.env