serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-llama-gemma4-q4km-200704.md

frank-zealot — llama.cpp CUDA, Gemma4-26B-A4B Q4_K_M, 1×200,704 (Dankclaw)

RETIRED 2026-10-07 — same day it went live (AGENTS.md Runbook Index). Superseded by frank-zealot-llama-qwen35-9b-q6k-2x200704 (Qwen3.5-9B Q6_K at 2 × 200,704 with exact KV); :7771 today runs frank-zealot-llama-prism-bonsai2-pq20 (llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-27B-PQ2_0.gguf, 2 × 200,704 q8_0 KV, MTP off). The payload stays at /home/dank/models/Gemma4-26B-A4B/ as the rollback (all three files present in the 2026-10-18 host census). This page keeps the Gemma4 KV-billing law (the attention_k_eq_v pair-billing measurement) and the drafter-cost boots, both of which outlive the model.

Payload

File	Path	Bytes
Main (Q4_K_M)	/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf	16,796,015,520 on disk; the engine reports 16,780,192,888
Vision (mmproj BF16)	/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf	1,194,827,776 B on disk (2026-10-18) = 1.11 GiB
MTP assistant (on disk, NOT loaded)	/home/dank/models/Gemma4-26B-A4B/mtp-gemma-4-26B-A4B-it.gguf	251,937,728 B on disk (2026-10-18) = 0.23 GiB

Repo: HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP (base google/gemma-4-26B-A4B-it). Downloaded 2026-10-07 by plain curl -fL (no token needed, ungated).

Shape

--ctx-size 200704 --parallel 1        # ONE conversation, Frank's 200k cap
--flash-attn on
--cache-type-k f16 --cache-type-v q8_0   # exact K, FP8-class V (q8_0)
--batch-size 4096 --ubatch-size 1024
--n-gpu-layers 99
--log-verbosity 4                     # debug: KV layout, buffers, graph capture
--spec-type none                      # MTP OFF (see below)
--mmproj <BF16 projector>
--jinja --reasoning auto --reasoning-format deepseek

Measured boot at q8_0 K + q8_0 V, ubatch 2048, MTP off (2026-10-07 morning): 21,890 MiB used / 2,235 MiB free (nvidia-smi). With MTP ON (the assistant loaded) the same shape was 23,806 / 319 — the drafter's context costs ~1.9 GiB because its KV is built against the target's full window (Llama Cpp Serving Sizing).

Deployed shape (2026-10-07 evening, Frank): K = f16, ubatch 1024, verbosity 4. Measured boot: 21,868 MiB used / 2,257 MiB free — i.e. the f16-K raise (+2,161 MiB of KV, see the accounting below) was paid for by the ubatch cut (2048 → 1024 returns ~616 MiB of compute buffer) plus slot-geometry variance. Exact K came essentially free. The verbosity-4 ledger (journal, llama_kv_cache lines): model 16,002.91 MiB (CUDA0) + 577.50 MiB (CPU-mapped), KV 3,001.25 MiB (5 full layers) + 306.25 MiB (25 SWA rings), compute buffers 926.39 + 150.63 MiB, output buffer 1.00 MiB.

Measured decode: 139.9 tok/s (/completion timings, temp 0, 256 tokens, deployed shape) — the ~4 B-active MoE is at home on the 3090. The model thinks: a 256-token budget on a short prompt can be consumed entirely by reasoning_content with empty content and finish_reason: length, so harness requests must budget for thinking (same law as the uncensored Swift payload). Vision probe on this build: solid-red 512 px PNG via the OpenAI image_url part → Red, prompt_tokens 151 (the image reached the model).

Decode is expected ubatch-invariant (Swift-era 3090 law; the 139.9 t/s above is at ubatch 1024 — one rung measured, not a ladder).

Architecture facts

Gemma4 MoE hybrid: 30 layers, 128 experts (8 active), n_embd 2816, 262,144 native window, n_vocab 262,144. Attention is hybrid per gemma4.attention.sliding_window_pattern (read from the file at load): 25 sliding-window layers (window 1024, n_head_kv = 8) + 5 full-attention layers (indices 5, 11, 17, 23, 29, n_head_kv = 2), head_dim 256 for K and V in both classes. The KV pool is TWO caches: the full-context cache scales with the 5 layers at the full 200,704 window; the SWA cache is a fixed 1024-token ring per sliding layer.

The rate law (measured, exact): bytes/token = n_kv_heads × 256 × 6.125 B per layer at f16 K + q8_0 V — attention_k_eq_v does NOT shrink the allocation: Vcur = Kcur is a compute-graph mapping, the cache reserves K and V slots and, with k_eq_v, reserves the pair twice (6.125 = 2 × (2 B f16 + 1.0625 B q8_0)). Verified against the boot: 5 × 2 × 256 × 6.125 = 15,680 B/token × 200,704 = 3,001.25 MiB ✓ and 25 × 1024 × 8 × 256 × 6.125 = 306.25 MiB ✓. At q8_0 K + q8_0 V the rate is 2.125 B/elem: the same shape costs 1,041 + 106 = 1,147 MiB, so the f16-K raise is +2,161 MiB. The --cache-type-v flag is honoured, not inert.

The llama.cpp master build at commit 5fc4f3c supports the arch (src/models/gemma4.cpp and LLM_ARCH_GEMMA4 = "gemma4" — both present in /home/dank/GitPackages/llama-master as read on 2026-10-18, src/llama-arch.cpp:59). Note: grep -c arch names on the packed binary returns 0 for EVERY arch (including the qwen3_5 it serves) — check the source tree, not the binary.

MTP law

MTP for Gemma4 is a SEPARATE file (mtp-gemma-4-26B-A4B-it.gguf, arch gemma4-assistant), loaded via --spec-draft-model beside --spec-type draft-mtp — unlike Swift, whose MTP head ships inside the main GGUF. The serve script gained SPEC_DRAFT_FILE for this. 2026-10-07 (Frank): MTP disabled — SPEC_TYPE=none strips both spec flags (flags move as a set). The assistant payload stays on disk; setting SPEC_TYPE=draft-mtp + SPEC_DRAFT_FILE re-enables it at the cost of ~1.9 GiB (boots at 319 MiB free).

Verification

• curl http://192.168.0.143:7771/health → {"status":"ok"}
• /v1/models reports n_ctx: 200704, ftype: Q4_K - Medium
• Vision probe: solid-color PNG ≥ 256 px via OpenAI image_url part, expect the exact color word.

Rollback

Revert to the Swift trio: MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/…, CTX_SIZE=229376, PARALLEL=2, in-file SPEC_TYPE=draft-mtp — the Swift payload remains on disk. See frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.