Frank
4b7de59dc8
The Qwen3.8-27B IQ3_S file carried its own MTP head; Gemma4's ships as a separate assistant file (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers) that loads via --model-draft and shares the target's last layers' KV. The draftless ngram-mod matcher chains behind it in the same --spec-type value — two context-backed drafters cannot share one process, so this chain is the only shape that carries both. KV is q8_0 (16 KiB/token/layer vs 32 at f16) and CTX_SIZE is unset so --fit sizes the pool against what the 7.4 GiB of payloads leave on the 24 GiB card. build-cuda was rebuilt with -DGGML_CUDA_FA_ALL_QUANTS=ON, which quantized KV on CUDA requires. The three seams that pinned the Qwen contract (MTP-in-file, the Qwen mmproj name, the f16 KV arithmetic) now pin the Gemma4 contract instead, and the deploy handler's presence check points at the new payload.
94 lines
4.3 KiB
Bash
94 lines
4.3 KiB
Bash
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
|
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
|
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
|
#
|
|
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
|
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
|
# cmake --build build-cuda --target llama-server -j
|
|
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
|
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
|
# CUDA (see the CUDA lesson in AGENTS.md).
|
|
#
|
|
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
|
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
|
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
|
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
|
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
|
# only a 1024-position window, which is why the KV pool is a fraction of
|
|
# the full-ctx x all-layers number.
|
|
#
|
|
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
|
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
|
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
|
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
|
# context loads the assistant model. The same --spec-type value chains the
|
|
# draftless ngram-mod matcher, which attaches unconditionally; two
|
|
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
|
# is the only legal shape that carries both.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): not modelled by hand -- CTX_SIZE is left
|
|
# unset and --fit sizes the KV pool (q8_0 K and V) against what the 6.87
|
|
# GiB of weights, the 0.16 GiB projector, the MTP sidecar's draft context
|
|
# and the compute buffers leave on the card. The engine prints the chosen
|
|
# context at startup; read it from the log, do not hard-code it back here.
|
|
#
|
|
# Vision: --mmproj loads the BF16 projector (11 tensors, 52M params; it
|
|
# also carries an audio encoder, unused by the fleet). The fleet's image
|
|
# path lives here, which is why halogen no longer sets HALOGEN_VISION_TOWER.
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
|
# FA_ALL_QUANTS=ON.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
|
|
|
# Model package. The MTP sidecar file is required for speculative decoding.
|
|
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
|
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
|
# what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Context: intentionally unset -- --fit decides it (see the sizing note).
|
|
# One slot; a second concurrent request queues behind the first.
|
|
PARALLEL=1
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical).
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
|
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
|
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# Every layer on the 3090, with flash attention (required for q8 KV).
|
|
NGL=999
|
|
FLASH_ATTN=on
|
|
|
|
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
|
# draftless ngram-mod matcher. Draft depth stays llama.cpp's default of 3.
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
|
|
# Greedy default (fleet contract); reasoning is extracted into
|
|
# reasoning_content with an unlimited budget (the model decides).
|
|
TEMPERATURE=0.0
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|