Frank
dbd764a493
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained window. Measured fit arithmetic on the 3090: 1 full-context slot idles at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed. A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit on purpose: fit maximises the TOTAL pool and would divide a smaller window across slots at parallel>1. The slot-grid seam now pins both the full-window division and the budget inequality so a future capacity bump fails pre-deploy, not in the boot allocation. Audio verified while measuring: the mmproj's audio encoder transcribes input_audio parts verbatim (JFK WAV sample). Header comments updated.
101 lines
4.6 KiB
Bash
101 lines
4.6 KiB
Bash
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
|
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
|
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
|
#
|
|
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
|
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
|
# cmake --build build-cuda --target llama-server -j
|
|
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
|
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
|
# CUDA (see the CUDA lesson in AGENTS.md).
|
|
#
|
|
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
|
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
|
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
|
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
|
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
|
# only a 1024-position window, which is why the KV pool is a fraction of
|
|
# the full-ctx x all-layers number.
|
|
#
|
|
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
|
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
|
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
|
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
|
# context loads the assistant model. The same --spec-type value chains the
|
|
# draftless ngram-mod matcher, which attaches unconditionally; two
|
|
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
|
# is the only legal shape that carries both.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
|
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
|
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
|
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
|
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
|
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
|
# which would shrink each slot below the full 262144 window once
|
|
# parallel > 1.
|
|
#
|
|
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
|
# params; it carries an audio encoder too). The fleet's image path lives
|
|
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
|
# WORKS: input_audio content parts transcribe verbatim (verified
|
|
# 2026-09-17 with a 16k mono WAV).
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
|
# FA_ALL_QUANTS=ON.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
|
|
|
# Model package. The MTP sidecar file is required for speculative decoding.
|
|
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
|
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
|
# what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
|
# pool; --parallel divides it). See the SIZING note above for the fit
|
|
# arithmetic; a fifth slot does not fit.
|
|
CTX_SIZE=1048576
|
|
PARALLEL=4
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical).
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
|
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
|
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# Every layer on the 3090, with flash attention (required for q8 KV).
|
|
NGL=999
|
|
FLASH_ATTN=on
|
|
|
|
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
|
# draftless ngram-mod matcher. Draft depth stays llama.cpp's default of 3.
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
|
|
# Greedy default (fleet contract); reasoning is extracted into
|
|
# reasoning_content with an unlimited budget (the model decides).
|
|
TEMPERATURE=0.0
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|