Frank
4d13333da7
Both were implicit build defaults (3 and 50); the throughput sweep needs to move them without editing the exec line each time. Defaults preserve current behavior exactly. Additive only.
105 lines
4.8 KiB
Bash
105 lines
4.8 KiB
Bash
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
|
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
|
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
|
#
|
|
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
|
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
|
# cmake --build build-cuda --target llama-server -j
|
|
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
|
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
|
# CUDA (see the CUDA lesson in AGENTS.md).
|
|
#
|
|
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
|
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
|
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
|
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
|
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
|
# only a 1024-position window, which is why the KV pool is a fraction of
|
|
# the full-ctx x all-layers number.
|
|
#
|
|
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
|
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
|
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
|
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
|
# context loads the assistant model. The same --spec-type value chains the
|
|
# draftless ngram-mod matcher, which attaches unconditionally; two
|
|
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
|
# is the only legal shape that carries both.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
|
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
|
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
|
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
|
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
|
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
|
# which would shrink each slot below the full 262144 window once
|
|
# parallel > 1.
|
|
#
|
|
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
|
# params; it carries an audio encoder too). The fleet's image path lives
|
|
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
|
# WORKS: input_audio content parts transcribe verbatim (verified
|
|
# 2026-09-17 with a 16k mono WAV).
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
|
# FA_ALL_QUANTS=ON.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
|
|
|
# Model package. The MTP sidecar file is required for speculative decoding.
|
|
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
|
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
|
# what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
|
# pool; --parallel divides it). See the SIZING note above for the fit
|
|
# arithmetic; a fifth slot does not fit.
|
|
CTX_SIZE=1048576
|
|
PARALLEL=4
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Batch geometry: llama.cpp's own defaults (2048 logical / 512 physical).
|
|
BATCH_SIZE=2048
|
|
UBATCH_SIZE=512
|
|
|
|
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
|
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
|
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
|
|
# Every layer on the 3090, with flash attention (required for q8 KV).
|
|
NGL=999
|
|
FLASH_ATTN=on
|
|
|
|
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
|
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
|
# explicit here for tuning (both were the implicit build defaults: 3 and 50;
|
|
# the Gemma4 sidecar carries 4 nextn layers).
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
SPEC_DRAFT_N_MAX=3
|
|
POLL_LEVEL=50
|
|
|
|
# Greedy default (fleet contract); reasoning is extracted into
|
|
# reasoning_content with an unlimited budget (the model decides).
|
|
TEMPERATURE=0.0
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|