---
type: reference
title: "frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M"
description: "The RETIRED frank-zealot runbook (2026-09-17 → 10-01) — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves."
tags:
  - llm
  - frank-zealot
  - dankclaw
  - llama.cpp
  - rtx3090
  - gemma
  - mtp
  - vision
timestamp: 2026-09-17
---

# frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

> **RETIRED 2026-09-17 → 2026-10-01** (AGENTS.md Runbook Index). It replaced
> [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) and
> [[frank-zealot-vllm-gemma4]] (the same-day vLLM experiment).
> Same port (7771), same alias (`frank-small` — this endpoint only carried the
> `frank-zealot` name from 2026-10-06/10-09), same client contract surface.
> :7771 today runs [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML
> fork build 10754, `Ternary-Bonsai-2-27B-PQ2_0.gguf`, 2 × 200,704 q8_0 KV, MTP
> off); the audio input this page documents left the fleet with this payload.

## Identity

- **Endpoint:** `frank-small` at `192.168.0.143:7771` (Dankclaw) —
  `git show e23df8b:scripts/frank-small.env` → `ALIAS=frank-small`.
- **Runtime:** mainline llama.cpp master `37b53fd`, CUDA build for sm_86 with
  `-DGGML_CUDA_FA_ALL_QUANTS=ON`
  (`/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server`).
  The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention
  collapses prefill ~12x on CUDA. Host check 2026-10-18: that binary reports
  `0.4.1-dev (build 1, commit 37b53fd)` and its `CMakeCache.txt` reads
  `GGML_CUDA_FA_ALL_QUANTS:BOOL=ON`.
- **Service shape:** root systemd unit `scripts/frank-zealot.service`
  (`User=dank`), launched by `scripts/serve-frank-zealot.sh` with tunables from
  `scripts/frank-zealot.env`.

## Shape

- **Model:** `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at
  Q4_K_M (6.87 GiB). Architecture `gemma4`: 48 layers, MHA with 16 kv heads
  x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer
  token embeddings (PLE) inside the file, and a sliding-window attention
  pattern (`attention.sliding_window = 1024`) — most layers cache only a
  1024-position window, which is why the KV pool is a fraction of the
  full-ctx x all-layers number.
- **Context:** four FULL 262,144-token slots (`CTX_SIZE=1048576`,
  `PARALLEL=4`). The explicit `CTX_SIZE` bypasses `--fit` on purpose: `--fit`
  maximises the TOTAL pool, which would shrink each slot below the full window
  once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra
  full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past
  the card. 4 is the measured max.
- **Placement:** the whole model in VRAM (`--gpu-layers 999`); the host CPU
  only tokenizes and samples (its CPU belongs to the halogen engine serving
  frank-dragoon).

## Spec method

The MTP head ships as a **separate** file (`mtp-gemma-4-12B-it.gguf`, arch
`gemma4-assistant`, 4 nextn layers, 0.24 GiB), loaded via `--model-draft`
with `--spec-type draft-mtp`. The same `--spec-type` value chains the
draftless **ngram-mod** matcher, which attaches unconditionally; two
context-backed drafters cannot share one process, so `draft-mtp,ngram-mod` is
the only legal shape that carries both. Draft depth 3 + poll 50 (measured:
depth 4 + poll 100 decoded WORSE at long context — the longer verification
round costs more attention re-read than the extra accepted token earns).

## KV

**q8_0** (`--cache-type-k q8_0 --cache-type-v q8_0`), with flash attention
(`--flash-attn on`, required for quantized KV). CPU-side cache capped at 2048
MiB (`CACHE_RAM=2048`) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the
3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so
the >12.8 GiB floor does not apply.

## Vision + audio

`--mmproj` loads the BF16 projector
(`mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf`, 11 tensors,
52M params), which also carries an audio encoder. The fleet's image path
lives here. Audio works: `input_audio` content parts transcribe verbatim
(verified 2026-09-17 with a 16k mono WAV). Fleet-wide audio ended with this
payload: no fleet endpoint takes audio input today.

## Payload

The main GGUF, the BF16 mmproj, and the MTP sidecar lived under
`/home/dank/models/Gemma4-12B-QAT`; they were staged to
`.AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/` on 2026-10-02 and
that stage dir is gone as of 2026-10-18, so the payload needs a re-fetch. The serve script refuses to start if any
is missing (a missing file is an unfinished download, and the unit's restart
policy would otherwise turn it into a crash loop that reads like a runtime
fault).

## Client contract

- **Alias:** `frank-small` (the `id` at `/v1/models`).
- **Reasoning:** extracted to `reasoning_content` (`--reasoning-format
  deepseek`, unlimited budget).
- **Tools:** supported (`--jinja` is pinned even though this build defaults
  it on, so a future build that flips the default cannot silently take tools
  away).
- **Vision + audio:** both supported in this era; audio is gone fleet-wide
  today (`input: [text, image]` — `~/.omp/agent/models.yml:214`).
- **Sampling:** greedy by default (fleet contract); a client that sends a
  temperature gets its own, and the harness applies per-model overrides from
  `models.yml`.

## Measured (max length)

At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024:
pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches
(2048) OOM the MTP draft context's pp compute buffer and must not be tried.

## Rollback

Payload and runtime for [[frank-zealot-qwen38-gsq-rco]] (ISTA-DASLab
Qwen3.8-27B-GSQ-RCO IQ3_S) are **not on disk any more** (the 2026-10-18 census
of `/home/dank/models/` has no GSQ-RCO directory and the era fetch script is
not in the tree) — that rollback re-downloads from ISTA-DASLab.
[[frank-zealot-vllm-gemma4]] is the same-day vLLM experiment; its weights are
gone too, though its derived image is still in rootless storage
(`localhost/vllm/vllm-openai:frank-small`, id `8dad46d79c46`).

# Citations

- Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
- llama.cpp: https://github.com/ggml-org/llama.cpp
- Serve script: `scripts/serve-frank-zealot.sh`; tunables: `scripts/frank-zealot.env`