Frank
24eaba1078
Swept on the 3090 at the deployed 4x262144 geometry, salted API probes: - ubatch 512: pp@254k ~918 t/s, tg@250k 90-204 (noisy) - ubatch 1024: pp@254k 962 t/s (+4.8%), tg@250k 257-302 - ubatch 2048: OOM — the MTP draft context needs ~3.3 GiB of pp compute buffer against 2.2 GiB headroom; do not try. - draft depth 4 + poll 100 decoded WORSE at 250k (230-257) than 3/50: the longer verification round re-reads more long-ctx attention than the extra accepted token earns. Kept 3/50, now with the evidence attached. Idle VRAM 23,300 MiB; all ubatch-1024 variants idled identically (compute buffers reserved at max graph size), so 8192 buys the deepest prefill window for free over 4096.
108 lines
5.2 KiB
Bash
108 lines
5.2 KiB
Bash
# Tunables for frank-small (HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-
|
|
# Balanced, Q4_K_M, plus the BF16 vision projector and the separate MTP
|
|
# sidecar) on mainline llama.cpp, Dankclaw's RTX 3090, since 2026-09-17.
|
|
#
|
|
# Runtime: llama.cpp master 37b53fd, CUDA build for sm_86:
|
|
# cmake -B build-cuda -DCMAKE_BUILD_TYPE=Release -DGGML_NATIVE=ON \
|
|
# -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=86 -DGGML_CUDA_FA_ALL_QUANTS=ON
|
|
# cmake --build build-cuda --target llama-server -j
|
|
# GGML_CUDA_FA_ALL_QUANTS is ON: this service runs q8_0 KV with flash
|
|
# attention, and without the flag quantized-KV prefill collapses ~12x on
|
|
# CUDA (see the CUDA lesson in AGENTS.md).
|
|
#
|
|
# Model: HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced at Q4_K_M
|
|
# (6.87 GiB). Architecture (GGUF arch gemma4): 48 layers, MHA with 16 kv
|
|
# heads x 512 head_dim, rope freq_base 1e6, final logit softcapping 30,
|
|
# per-layer token embeddings (PLE) inside the file, and a sliding-window
|
|
# attention pattern (attention.sliding_window = 1024) -- most layers cache
|
|
# only a 1024-position window, which is why the KV pool is a fraction of
|
|
# the full-ctx x all-layers number.
|
|
#
|
|
# MTP: the Multi-Token Prediction head ships as a SEPARATE file
|
|
# (mtp-gemma-4-12B-it.gguf, arch gemma4-assistant, 4 nextn layers, 0.24
|
|
# GiB). Unlike the Qwen3.8 -mtp build (head inside the main file), this one
|
|
# is passed via --model-draft with --spec-type draft-mtp, so the draft
|
|
# context loads the assistant model. The same --spec-type value chains the
|
|
# draftless ngram-mod matcher, which attaches unconditionally; two
|
|
# context-backed drafters cannot share one process, so draft-mtp,ngram-mod
|
|
# is the only legal shape that carries both.
|
|
#
|
|
# SIZING (RTX 3090, 24576 MiB): CTX_SIZE = PARALLEL x 262144 -- four
|
|
# FULL-window slots, measured to fit: the 4-slot boot idles at 22,346
|
|
# MiB and each extra full-context slot costs 3,450 MiB (measured 1-slot
|
|
# 11,996 -> 4-slot 22,346). A fifth slot projects to ~25.8 GiB, past the
|
|
# card, and the KV pool allocates upfront -- so 4 is the max. Explicit
|
|
# CTX_SIZE bypasses --fit on purpose: --fit maximises the TOTAL pool,
|
|
# which would shrink each slot below the full 262144 window once
|
|
# parallel > 1.
|
|
#
|
|
# Vision + audio: --mmproj loads the BF16 projector (11 tensors, 52M
|
|
# params; it carries an audio encoder too). The fleet's image path lives
|
|
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER. Audio
|
|
# WORKS: input_audio content parts transcribe verbatim (verified
|
|
# 2026-09-17 with a 16k mono WAV).
|
|
#
|
|
# Sampling: greedy by default (the fleet contract); a client that sends a
|
|
# temperature gets its own, and the harness applies per-model overrides
|
|
# from models.yml.
|
|
#
|
|
# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE (HIP
|
|
# reads those bare vars on the ROCm hosts -- see the 2026-08-13 incident in
|
|
# AGENTS.md). This box is CUDA, but the rule keeps env files portable.
|
|
|
|
# llama.cpp runtime binary: mainline master 37b53fd, CUDA sm_86,
|
|
# FA_ALL_QUANTS=ON.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama.cpp-mainline/build-cuda/bin/llama-server
|
|
|
|
# Model package. The MTP sidecar file is required for speculative decoding.
|
|
MODEL_DIR=/home/dank/models/Gemma4-12B-QAT
|
|
MODEL_FILE=Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_PATH=/home/dank/models/Gemma4-12B-QAT/mmproj-Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
DRAFT_FILE=mtp-gemma-4-12B-it.gguf
|
|
|
|
# Fleet identity and network. 7771 is this service's own endpoint on
|
|
# Dankclaw; 7770 stays frank-large's. The alias is the id at /v1/models and
|
|
# what clients ask for -- the harness entry in models.yml names it.
|
|
ALIAS=frank-small
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
|
|
# Four FULL 262144-token contexts, one per slot (CTX_SIZE is the total
|
|
# pool; --parallel divides it). See the SIZING note above for the fit
|
|
# arithmetic; a fifth slot does not fit.
|
|
CTX_SIZE=1048576
|
|
PARALLEL=4
|
|
CACHE_TYPE_K=q8_0
|
|
CACHE_TYPE_V=q8_0
|
|
|
|
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
|
|
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):
|
|
# ubatch 512 -> pp 915 t/s, tg@250k 90-204; ubatch 1024 -> pp 962,
|
|
# tg 257-302. Larger ubatches (2048) OOM the MTP draft context's pp
|
|
# compute buffer (~3.3 GiB against 2.2 GiB headroom) and must not be
|
|
# tried. Idle VRAM is 23,300 MiB at this geometry.
|
|
BATCH_SIZE=8192
|
|
UBATCH_SIZE=1024
|
|
|
|
# CPU threads: this service runs entirely on the GPU (NGL=999), so threads
|
|
# cover tokenization, sampling and the HTTP layer only. Deliberately modest
|
|
# -- this host's CPU is shared with halogen's engine, which is CPU-hungry.
|
|
NUM_THREADS=8
|
|
THREAD_BATCH=8
|
|
# Speculative decoding: the MTP sidecar (draft context) chained with the
|
|
# draftless ngram-mod matcher. Draft depth and busy-wait level are made
|
|
# explicit here (both were the implicit build defaults: 3 and 50).
|
|
# Measured at 250k ctx: depth 4 + poll 100 DECODED WORSE (230-257 t/s)
|
|
# than depth 3 + poll 50 (257-302 t/s) — the longer verification round
|
|
# costs more attention re-read at long context than the extra accepted
|
|
# token earns. Keep 3/50.
|
|
SPEC_TYPE=draft-mtp,ngram-mod
|
|
SPEC_DRAFT_N_MAX=3
|
|
POLL_LEVEL=50
|
|
|
|
# Greedy default (fleet contract); reasoning is extracted into
|
|
# reasoning_content with an unlimited budget (the model decides).
|
|
TEMPERATURE=0.0
|
|
REASONING_FORMAT=deepseek
|
|
REASONING_BUDGET=-1
|