Frank
b51e078e36
Verbosity 4 paid for itself: the engine's own llama_kv_cache lines give the exact accounting — the cache bills the K+V pair TWICE under attention_k_eq_v (6.125 B/elem at f16 K + q8_0 V), reproducing 3,001.25 MiB (5 full layers at n_head_kv 2) + 306.25 MiB (25 SWA rings at n_head_kv 8, window 1024) to the byte. The earlier "bill K-only" note was wrong; full layers have 2 KV heads, not 8. Deployed boot: 21,868 used / 2,257 free — the f16-K raise (+2,161 MiB KV) is funded by the ubatch 2048 -> 1024 cut (~616 MiB of compute buffer) plus variance, so exact K came essentially free. Measured decode 139.9 t/s temp 0; vision probe returns Red at prompt_tokens 151.
47 lines
2.1 KiB
Bash
47 lines
2.1 KiB
Bash
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
|
# HauhauCS/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-MTP
|
|
# (Q4_K_M) on the RTX 3090, :7771.
|
|
#
|
|
# 2026-10-07 (Frank): swap from Swift-1.5-Qwen3.8-27B RCO-IQ3_XXS to
|
|
# Gemma4-26B-A4B Q4_K_M, ONE 200,704 context, f16 K + q8_0 V,
|
|
# MTP DISABLED (SPEC_TYPE=none), log verbosity 4.
|
|
#
|
|
# Architecture (read from the file's own header at load): gemma4 MoE — 30
|
|
# layers, 128 experts (8 active), sliding_window_pattern = 25 sliding
|
|
# (window 1024, n_head_kv 8) + 5 full-attention (layers 5/11/17/23/29,
|
|
# n_head_kv 2), head_dim 256, attention_k_eq_v, native 262,144 window.
|
|
#
|
|
# KV rate law (measured, exact): the cache bills the K+V pair TWICE under
|
|
# attention_k_eq_v — 6.125 B/elem at f16 K + q8_0 V, 2.125 B/elem at
|
|
# q8_0/q8_0. Full cache = 5 x 2 x 256 x rate x ctx; SWA cache = 25 x 1024 x
|
|
# 8 x 256 x rate. There is no true fp8 KV dtype in this build; q8_0 is the
|
|
# FP8 class.
|
|
#
|
|
# MTP is OFF: SPEC_TYPE=none strips --spec-type AND --spec-draft-model
|
|
# (spec flags move as a set). The assistant file
|
|
# (mtp-gemma-4-26B-A4B-it.gguf, 0.252 GiB) stays on disk as the drafter
|
|
# payload if MTP is ever re-enabled. Measured VRAM: MTP ON booted
|
|
# 23,806 used / 319 free; MTP OFF boots 21,890 used / 2,235 free.
|
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
MODEL_FILE=/home/dank/models/Gemma4-26B-A4B/Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf
|
|
MMPROJ_FILE=/home/dank/models/Gemma4-26B-A4B/mmproj-Gemma4-26B-A4B-QAT-Uncensored-HauhauCS-Balanced-BF16.gguf
|
|
ALIAS=frank-secondary
|
|
BIND_ADDR=0.0.0.0
|
|
LISTEN_PORT=7771
|
|
CTX_SIZE=200704
|
|
PARALLEL=1
|
|
BATCH_SIZE=4096
|
|
UBATCH_SIZE=1024
|
|
CACHE_TYPE_K=f16
|
|
CACHE_TYPE_V=q8_0
|
|
SPEC_TYPE=none
|
|
NGL=99
|
|
|
|
# 2026-10-07 (Frank): verbosity 4 (debug — KV layout, buffers, graph
|
|
# capture), ubatch 1024, K raised to f16 (V stays q8_0). MEASURED BOOT of
|
|
# this exact shape: 21,868 MiB used / 2,257 MiB free — KV 3,001.25 (full) +
|
|
# 306.25 (SWA) MiB (+2,161 over q8_0 K), compute buffers 926.39 + 150.63
|
|
# (the 2048 rung cost ~616 MiB more, which is what funded the raise). The
|
|
# q8_0-K/ubatch-2048 predecessor booted 21,890 / 2,235.
|
|
VERBOSITY=4
|