# frank-secondary — llama.cpp (master, CUDA sm_86) serving
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
# (Q4_K_M) on the RTX 3090, :7771.
#
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
#
# Architecture (read from the file's own header at load): gemma4 MoE — 30
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
#
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
# FP8 class.
#
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
# (spec flags move as a set). The assistant file
# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter
# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted
# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free.
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
ALIAS=frank-secondary
BIND_ADDR=0.0.0.0
LISTEN_PORT=7771
CTX_SIZE=200704
PARALLEL=1
BATCH_SIZE=4096
UBATCH_SIZE=1024
CACHE_TYPE_K=f16
CACHE_TYPE_V=q8_0
SPEC_TYPE=none
NGL=99

# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
VERBOSITY=4