dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
2427fc448fe961916f18285288de5a50f78621d8
serverAppConfigs/serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md
T
Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages) 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
2026-10-18 13:15:43 -07:00

6.5 KiB
Raw Blame History

type, title, description, tags, timestamp
type title description tags timestamp
reference frank-zealot on Dankclaw — llama.cpp, Gemma4-12B QAT Q4_K_M The RETIRED frank-zealot runbook (2026-09-17 → 10-01) — mainline llama.cpp on Dankclaw's RTX 3090 serving HauhauCS Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M with a separate MTP assistant sidecar chained behind ngram-mod, q8_0 KV, and --fit sizing the full 262k window: identity, payload, KV arithmetic, the spec-decode chain, deployed flags, the client contract, and what to check when it misbehaves.
llm
frank-zealot
dankclaw
llama.cpp
rtx3090
gemma
mtp
vision
2026-09-17

frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

RETIRED 2026-09-17 → 2026-10-01 (AGENTS.md Runbook Index). It replaced frank-zealot-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) and frank-zealot-vllm-gemma4 (the same-day vLLM experiment). Same port (7771), same alias (frank-small — this endpoint only carried the frank-zealot name from 2026-10-06/10-09), same client contract surface. :7771 today runs frank-zealot-llama-prism-bonsai2-pq20 (llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-27B-PQ2_0.gguf, 2 × 200,704 q8_0 KV, MTP off); the audio input this page documents left the fleet with this payload.

Identity

  • Endpoint: frank-small at 192.168.0.143:7771 (Dankclaw) — git show e23df8b:scripts/frank-small.env → ALIAS=frank-small.
  • Runtime: mainline llama.cpp master 37b53fd, CUDA build for sm_86 with -DGGML_CUDA_FA_ALL_QUANTS=ON (/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server). The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention collapses prefill ~12x on CUDA. Host check 2026-10-18: that binary reports 0.4.1-dev (build 1, commit 37b53fd) and its CMakeCache.txt reads GGML_CUDA_FA_ALL_QUANTS:BOOL=ON.
  • Service shape: root systemd unit scripts/frank-zealot.service (User=dank), launched by scripts/serve-frank-zealot.sh with tunables from scripts/frank-zealot.env.

Shape

  • Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (6.87 GiB). Architecture gemma4: 48 layers, MHA with 16 kv heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer token embeddings (PLE) inside the file, and a sliding-window attention pattern (attention.sliding_window = 1024) — most layers cache only a 1024-position window, which is why the KV pool is a fraction of the full-ctx x all-layers number.
  • Context: four FULL 262,144-token slots (CTX_SIZE=1048576, PARALLEL=4). The explicit CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, which would shrink each slot below the full window once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past the card. 4 is the measured max.
  • Placement: the whole model in VRAM (--gpu-layers 999); the host CPU only tokenizes and samples (its CPU belongs to the halogen engine serving frank-dragoon).

Spec method

The MTP head ships as a separate file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 GiB), loaded via --model-draft with --spec-type draft-mtp. The same --spec-type value chains the draftless ngram-mod matcher, which attaches unconditionally; two context-backed drafters cannot share one process, so draft-mtp,ngram-mod is the only legal shape that carries both. Draft depth 3 + poll 50 (measured: depth 4 + poll 100 decoded WORSE at long context — the longer verification round costs more attention re-read than the extra accepted token earns).

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), with flash attention (--flash-attn on, required for quantized KV). CPU-side cache capped at 2048 MiB (CACHE_RAM=2048) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the 3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so the >12.8 GiB floor does not apply.

Vision + audio

--mmproj loads the BF16 projector (mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf, 11 tensors, 52M params), which also carries an audio encoder. The fleet's image path lives here. Audio works: input_audio content parts transcribe verbatim (verified 2026-09-17 with a 16k mono WAV). Fleet-wide audio ended with this payload: no fleet endpoint takes audio input today.

Payload

The main GGUF, the BF16 mmproj, and the MTP sidecar lived under /home/dank/models/Gemma4-12B-QAT; they were staged to .AGENTTRASH/20261002-retired-llm-payloads/Gemma4-12B-QAT/ on 2026-10-02 and that stage dir is gone as of 2026-10-18, so the payload needs a re-fetch. The serve script refuses to start if any is missing (a missing file is an unfinished download, and the unit's restart policy would otherwise turn it into a crash loop that reads like a runtime fault).

Client contract

  • Alias: frank-small (the id at /v1/models).
  • Reasoning: extracted to reasoning_content (--reasoning-format deepseek, unlimited budget).
  • Tools: supported (--jinja is pinned even though this build defaults it on, so a future build that flips the default cannot silently take tools away).
  • Vision + audio: both supported in this era; audio is gone fleet-wide today (input: [text, image] — ~/.omp/agent/models.yml:214).
  • Sampling: greedy by default (fleet contract); a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Measured (max length)

At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024: pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches (2048) OOM the MTP draft context's pp compute buffer and must not be tried.

Rollback

Payload and runtime for frank-zealot-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) are not on disk any more (the 2026-10-18 census of /home/dank/models/ has no GSQ-RCO directory and the era fetch script is not in the tree) — that rollback re-downloads from ISTA-DASLab. frank-zealot-vllm-gemma4 is the same-day vLLM experiment; its weights are gone too, though its derived image is still in rootless storage (localhost/vllm/vllm-openai:frank-small, id 8dad46d79c46).

Citations

Reference in New Issue View Git Blame Copy Permalink