0 Fork 0

Files

T

Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot

The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.

2026-10-10 17:03:14 -07:00

5.3 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

frank-zealot — llama.cpp Gemma4-12B QAT Q4_K_M (MTP + ngram-mod)

ACTIVE since 2026-09-17. Replaces frank-zealot-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S, now the rollback target) and frank-zealot-vllm-gemma4 (the same-day vLLM experiment, also the rollback target). Same port (7771), same alias (frank-zealot), same client contract surface.

Identity

Endpoint: frank-zealot at 192.168.0.143:7771 (Dankclaw).
Runtime: mainline llama.cpp master 37b53fd, CUDA build for sm_86 with -DGGML_CUDA_FA_ALL_QUANTS=ON (/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server). The FA_ALL_QUANTS flag is required: without it, q8_0 KV + flash attention collapses prefill ~12x on CUDA.
Service shape: root systemd unit scripts/frank-zealot.service (User=dank), launched by scripts/serve-frank-zealot.sh with tunables from scripts/frank-zealot.env.

Shape

Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M (6.87 GiB). Architecture gemma4: 48 layers, MHA with 16 kv heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30, per-layer token embeddings (PLE) inside the file, and a sliding-window attention pattern (attention.sliding_window = 1024) — most layers cache only a 1024-position window, which is why the KV pool is a fraction of the full-ctx x all-layers number.
Context: four FULL 262,144-token slots (CTX_SIZE=1048576, PARALLEL=4). The explicit CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool, which would shrink each slot below the full window once parallel > 1. The 4-slot boot idles at 22,346 MiB; each extra full-context slot costs 3,450 MiB, and a fifth projects to ~25.8 GiB — past the card. 4 is the measured max.
Placement: the whole model in VRAM (--gpu-layers 999); the host CPU only tokenizes and samples (its CPU belongs to the halogen engine serving frank-dragoon).

Spec method

The MTP head ships as a separate file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24 GiB), loaded via --model-draft with --spec-type draft-mtp. The same --spec-type value chains the draftless ngram-mod matcher, which attaches unconditionally; two context-backed drafters cannot share one process, so draft-mtp,ngram-mod is the only legal shape that carries both. Draft depth 3 + poll 50 (measured: depth 4 + poll 100 decoded WORSE at long context — the longer verification round costs more attention re-read than the extra accepted token earns).

KV

q8_0 (--cache-type-k q8_0 --cache-type-v q8_0), with flash attention (--flash-attn on, required for quantized KV). CPU-side cache capped at 2048 MiB (CACHE_RAM=2048) to give back ~6 GiB of LPDDR — Gemma4 runs fully on the 3090, so the CPU cache is not the hot path, and it is not a DeltaNet model so the >12.8 GiB floor does not apply.

Vision + audio

--mmproj loads the BF16 projector (mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf, 11 tensors, 52M params), which also carries an audio encoder. The fleet's image path lives here. Audio works: input_audio content parts transcribe verbatim (verified 2026-09-17 with a 16k mono WAV).

Payload

The main GGUF, the BF16 mmproj, and the MTP sidecar live under /home/dank/models/Gemma4-12B-QAT. The serve script refuses to start if any is missing (a missing file is an unfinished download, and the unit's restart policy would otherwise turn it into a crash loop that reads like a runtime fault).

Client contract

Alias: frank-zealot (the id at /v1/models).
Reasoning: extracted to reasoning_content (--reasoning-format deepseek, unlimited budget).
Tools: supported (--jinja is pinned even though this build defaults it on, so a future build that flips the default cannot silently take tools away).
Vision + audio: supported.
Sampling: greedy by default (fleet contract); a client that sends a temperature gets its own, and the harness applies per-model overrides from models.yml.

Measured (max length)

At 254k-token prefill / 253k-ctx decode, 4 slots, q8_0 KV, ubatch 1024: pp ~965 t/s, tg 257–302 t/s, idle 23,300 MiB of 24,576. Larger ubatches (2048) OOM the MTP draft context's pp compute buffer and must not be tried.

Rollback

Payload and runtime for frank-zealot-qwen38-gsq-rco (ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S) remain intact on disk. frank-zealot-vllm-gemma4 is the same-day vLLM experiment (also a rollback target).

Citations

Model + mmproj + MTP sidecar: https://huggingface.co/HauhauCS
llama.cpp: https://github.com/ggml-org/llama.cpp
Serve script: scripts/serve-frank-zealot.sh; tunables: scripts/frank-zealot.env

5.3 KiB Raw Blame History Unescape Escape